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EXECUTIVE  SUMMARY 


The  FAA  has  been  involved  in  the  development  and 
implementation  of  training  programs  for  aeronautical 
decision  making  (ADM)  since  1973.  The  first  genera¬ 
tion  ADM  materials  focused  on  the  development  of  an 
awareness  of  hazardous  attitudes.  The  students  were 
introduced  to  the  subject  of  decision  making  followed  by 
discussions  of  hazardous  thoughts,  error  chains,  and  risk 
assessment.  T raining  manuals  were  provided  with  ADM 
applications  to  student,  instructor,  commercial,  instru¬ 
ment,  and  helicopter  pilots.  An  application  of  ADM  to 
multi-person  crews  was  offered  in  a  crew  resource  man¬ 
agement  (CRM)  training  manual  as  well. 

The  current  project  was  composed  of  four  studies 
designed  to  formulate  a  model  of  the  expert  pilot  that 
could  be  tested  and  used  in  developing  intervention 
strategies.  Each  study  in  turn  contributed  to  an  evolving 
model  of  the  expert  pilot.  It  should  be  noted  that  these 
were  exploratory  studies,  using  small  numbers  of  sub¬ 
jects  in  an  effort  to  define  the  areas  of  interest.  Conse- 
quendy,  statistical  tests  and  other  procedures  which  may 
be  utilized  with  more  rigorous  studies  involving  larger 
samples  of  pilots  are  not  appropriate  and  are  not  re¬ 
ported.  Nevertheless,  the  qualitative  information  ob¬ 
tained  from  these  studies  may  be  of  substantial  benefit  in 
shaping  our  thinking  about  how  pilots  make  decisions 
and  in  plotting  our  future  course  of  research. 

During  the  first  study,  pilots  were  interviewed  to 
identify  and  compile  characteristics  of  the  expert  pilot. 
The  responses  to  the  initial  questionnaire  were  compiled 
and  characteristics  of  the  expert  pilot  were  placed  into 
categories.  These  initial  categories  were  skills,  procedural 
knowledge,  learning  and  performance  strategies,  confi¬ 
dence,  and  motivation. 

In  the  second  study,  structured  interviews  were  con¬ 
ducted  with  thirty  pilots  who  met  our  initial  character¬ 
ization  of  expert.  From  these  interviews,  a  preliminary 
definition  of  an  expert  pilot  emerged:  “One  who  is  highly 
motivated,  confident  (but  not  overconfident),  has  supe¬ 
rior  learning  and  performance  skills,  applies  those  skills 
in  a  changing  environment,  and  possesses  a  type  of 
judgment  described  by  many  as  ‘magic’  or  ‘natural’.” 
Some  pilots  described  the  expert  as  one  who  becomes  “a 


part  of  the  machine  and  flows  within  the  ‘flying  space’.” 
From  this  second  study,  a  new  list  of  distinguishing 
characteristics  was  identified. 

The  third  study  evaluated  the  candidate  definition  of 
expert  pilot  in  three  types  of  aircraft  from  the  general 
aviation  mid-altitude  flying  domain:  Beech  P-Baron, 
Piper  Malibu,  and  Cessna  P-210.  National  Transporta¬ 
tion  Safety  Board  accident  reports  were  reviewed  for 
these  three  types  of  aircraft.  Contributing  factors  in  the 
accidents  were  used  to  create  a  realistic  scenario  with  a  set 
of  seven  flight  events  for  use  in  a  flight  simulation  type 
verbal  protocol  analysis.  Characteristics  of  each  of  the 
events  were  put  in  outline  form  under  the  general 
headings:  knowledge,  skills,  and  mental  models,  at  three 
levels:  novice,  competent,  and  expert.  After  this  study, 
the  modeltooktheformofapreliminarymodelofthe  expert 
pilot  in  oudine  form  with  the  following  major  headings: 
knowledge,  skills  or  abilities,  behavior,  and  motivation. 

In  the  fourth  study,  the  scenario  developed  in  Study 
3  was  presented  to  subjects  and  their  responses  were 
recorded.  The  recordings  were  transcribed  and  encoded 
into  specific  categories  for  data  analysis.  The  categories 
chosen  roughly  corresponded  to  the  major  points  of  the 
model  of  the  expert  pilot  developed  in  Study3.  Frequen¬ 
cies  of  subject  responses  in  each  category  were  tabulated 
for  later  analysis.  T rends  in  these  data  indicate  that  pilots 
who  achieved  better  overall  flight  results  could  be  differ¬ 
entiated  from  those  who  were  less  successful  in  three 
ways:  (1)  they  seek  more  quality  information  in  a  more 
timely  manner,  (2)  they  make  more  progressive  decisions 
to  solve  a  problem,  and  (3)  they  communicate  more 
readily  with  all  available  resources.  Subjective  analyses  of 
the  transcripts  show  that  pilots  do  indeed  have  different 
methods  and  styles  for  solving  fairly  common  flying 
situations,  and  these  methods  are  not  related  to  the  total 
flight  time  or  years  flying.  These  tentative  findings  are, 
of  course,  subject  to  verification  using  larger  samples  in 
more  rigorous,  controlled  situations.  Such  replications 
using  similar  scenarios  in  flight  simulators  are  planned 
which  will  explore  more  completely  the  pilot  perfor¬ 
mance  model  parameters  that  these  exploratory  studies 
have  suggested  are  important. 
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A  NEW  APPROACH  TO  AERONAUTICAL  DECISION  MAKING: 

THE  EXPERTISE  METHOD 


INTRODUCTION 

The  study  of  expertise  seeks  to  understand  and 
account  for  what  distinguishes  outstanding  individu¬ 
als  in  a  domain  from  less  outstanding  individuals  in 
that  domain,  as  well  as  from  people  in  general.  The 
approach  focuses  on  outstanding  behavior  that  can  be 
attributed  to  relatively  stable,  learned  characteristics 
of  the  relevant  individuals.  The  classical  expertise 
literature  suggests  that  aggregation  of  experience  (e.g., 
ten  years  of  full  time  work  in  a  domain)  is  the  single 
most  important  factor  in  the  acquisition  of  expertise 
(Chase  and  Simon,  1973).  In  most  cases,  no  distinc¬ 
tion  is  made  concerning  the  type  of  experience  the 
expert  has  had  as  long  as  it  is  acquired  roughly  in  the 
domain  of  interest.  On  the  other  hand,  Ericsson  & 
Smith  (1991)  found  several  studies  (Libby  and 
Frederick,  1989,  Gustafson,  1963)  revealing  that 
people  with  many  years  of  experience  in  a  domain 
performed  only  slightly  better  than  those  just  coming 
out  of  training.  They  conclude  that  the  greatest  amount 
of  improvement  occurs  in  training,  not  as  a  result  of 
years  of  experience. 

Building  cognitive  models  using  the  expertise  ap¬ 
proach  involves  three  steps:  1)  Identifying  representa¬ 
tive  tasks  that  capture  the  essence  of  superior 
performance  in  a  specific  domain,  2)  Detailed  analysis 
of  the  superior  performance  through  several  methods 
including  verbal  reports  during  performance  of  the 
tasks,  and  3)  Efforts  to  account  for  the  acquisition  of 
the  characteristics  and  cognitive  structures  found  to 
mediate  superior  performances  of  experts.  For  a  more 
complete  description  of  the  expertise  approach,  see 
Downs,  Jensen,  and  Chubb  (in  preparation). 

In  aviation  the  accident  record  shows  that  it  is  in 
the  area  of  cognitive  skills  where  pilots  most  often  fail. 
Although  general  aviation  can  point  to  some  success¬ 
ful  attempts  (Fox,  1991;  Diehl,  1992),  deliberate 
teaching  of  judgment  skills  is  rare.  Crew  resource 
management  (CRM)  programs  in  the  airline  environ¬ 


ment,  which  are  closely  associated  with  aeronautical 
decision  making  (ADM)  training,  seem  to  be  having 
a  useful  effect,  but  assessment  strategies  are  lacking, 
making  it  difficult  to  gage  the  effects  with  certainty.  It 
seems  clear  that  the  early  approaches  to  ADM  training 
may  have  reached  a  plateau.  The  time  seems  right  for 
a  re-examination  of  the  basic  approach  to  ADM  and 
perhaps  to  propose  new  intervention  strategies. 

The  objective  of  this  research  effort  is  to  develop 
new  models  of  ADM  to  provide  a  better  understand¬ 
ing  of  the  concept  in  the  general  aviation  domain. 
From  these  models,  new  intervention  strategies  will 
be  developed,  tested  and  validated.  The  ultimate 
objective  is  safer  general  aviation  operations.  The 
present  research  effort  consists  of  a  series  of  studies  to 
develop  models  of  the  mid-altitude  general  aviation 
pilot.  In  these  studies  the  sub-goals  are  to  1)  deter¬ 
mine  the  distinguishing  qualities  of  expert  aviators,  2) 
assess  the  processes  by  which  they  have  acquired  their 
expertise,  and  3)  create  a  training  and  evaluation 
system  to  bring  the  competent  pilot  closer  to  the 
expert.  Whereas  most  previous  research  on  ADM  has 
attempted  to  change  pilot  attitudes,  the  present  study 
was  focused  on  understanding  the  thought  processes 
of  the  general  aviation  pilot.  Through  the  use  of 
several  methods  of  cognitive  analysis  (performance 
analysis,  expert-novice  comparisons,  and  verbal  pro¬ 
tocol  analysis),  an  attempt  was  made  to  distinguish 
the  qualities  of  the  expert  and  the  competent  general 
aviation  pilot.  Through  this  process,  a  cognitive  model 
was  developed.  This  model  will  be  used  to  develop  a 
new  intervention  strategy  for  teaching  these  skills. 

The  domain  selected  for  this  study  was  a  subset  of 
general  aviation  pilots  who  fly  the  Cessna  P-210 
(Centurion),  the  Beechcraft  58-P  (Baron),  and  the 
Piper  PA-48  (Malibu).  These  aircraft  are  considered 
complex,  fairly  high-performance  single-engine  and 
twin-engine  (Baron)  aircraft  and  are  usually  flown  by 
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a  single  pilot  often  in  the  most  complex  airspace  and 
in  all  weather  conditions.  For  these  reasons,  the  flying 
task  for  pilots  in  this  domain  maybe  as  difficult  as  any 
in  civilian  aviation.  To  add  to  the  difficulty  of  the 
task,  many  of  the  pilots  of  these  aircraft  have  other 
primary  professions  (e.g.,  doctors,  lawyers,  business¬ 
men,  etc.)  and  do  not  fly  very  often  and,  therefore, 
may  not  be  as  proficient  as  pilots  who  make  flying 
their  primary  profession.  These  selected  models  of 
aircraft  have  more  complex  systems  and  generally  fly 
high-altitude  flight  envelopes  compared  to  most  gen¬ 
eral  aviation  light  aircraft  (Landsburg,  1992).  The 
National  Transportation  Safety  Board  (NTSB)  re¬ 
ports  of  accidents  for  these  types  of  aircraft  were 
examined  from  the  beginning  of  their  respective  pro¬ 
duction  programs  (Cessna  -  1978,  Piper  -  1984, 
Beechcraft  -  1976).  One  of  the  advantages  of  research 
in  this  particular  domain  is  that  pilot  subjects  are 
available  who  cover  the  whole  range  of  expertise. 
Using  the  expertise  approach  taps  into  their  thinking 
in  such  a  way  that  many  of  the  results  may  be  generaliz- 
able  to  the  entire  general  aviation  pilot  population. 

Identification  and  selection  of  subjects  varied  for 
each  study  in  this  program.  Subjects  for  each  study 
were  selected  because  they  represented  the  segment  of 
the  pilot  population  being  studied  and  because  they 
had  the  level  of  expertise  being  sought.  Some  indi¬ 
viduals  participated  in  more  than  one  phase  of  the 
experiment.  This  occurred  when  a  pilot  was  initially 
selected  for  the  initial  and/or  the  secondary  interview 
and  had  the  prerequisite  qualifications  for  other  phases 
(e.g.  flying  time  in  one  of  the  three  aircraft  represent¬ 
ing  our  domain). 

To  accomplish  the  primary  objective  of  this  study, 
to  build  a  model  of  the  expert  general  aviation  aviator, 
several  studies  were  conducted  including  1)  literature 
reviews  of  the  decision  making,  expertise,  and  judg¬ 
ment  literature,  2)  a  series  of  studies  using  unstruc¬ 
tured  interviews,  structured  interviews,  cognitive  task 
analysis,  and  verbal  protocol  analysis  approach,  and  3) 
a  modeling  design  effort.  The  model  of  the  expert 
general  aviation  pilot  evolved  following  each  study, 
first  taking  the  form  of  a  definition  of  expertise, 
second,  a  list  of  distinguishing  characteristics  of  ex¬ 
pert,  third,  a  taxonomy  of  expert  characteristics,  and 
finally,  a  model  for  creating  the  expert  aviator. 


DECISION  MAKING  LITERATURE 
REVIEW 

Background 

There  is  a  very  large  body  of  literature  on  human 
decision  making;  however,  most  of  it  is  not  relevant  to 
the  present  study.  Our  review  of  the  literature  was 
restricted  to  those  studies  that  support  our  effort  to 
build  models  of  aviator  decision  making.  Our  purpose 
was  to  determine  what  others  have  done  in  the  field  of 
aeronautical  decision  making  and  expertise  in  avia¬ 
tion  and  other  domains  and  to  build  on  the  knowledge 
base  in  this  study.  More  complete  descriptions  of  the 
expertise  and  modeling  research  literature  are  pro¬ 
vided  in  Downs,  Jensen,  and  Chubb,  (in  preparation) 
and  Chubb  and  Jensen  (in  preparation). 

Early  Judgment  Research 

As  early  as  1919  (Henmon,  1919)  psychological 
tests  relating  to  complex  psychomotor  reaction  and 
attention  were  used  routinely  during  induced  periods 
of  low  oxygen  in  the  belief  that  the  results  indirectly 
revealed  certain  psychological  characteristics.  During 
the  later  stages  of  oxygen  deprivation,  there  were 
unusual  emotional  outbursts  of  hilarity  or  anger  that 
many  thought  revealed  basic  aspects  of  the  pilot’s 
emotional  life  or  personality.  In  1921,  researchers 
(Dockeray  and  Issacs,  1921)  were  looking  at  judg¬ 
ment  as  part  of  psychological  research  in  aviation.  In 
a  study  of  the  physiological  and  psychological  charac¬ 
teristics  of  civil  airline  pilots  (McFarland,  Graybiel, 
Liljerfgrahtz,  and  Tuttle,  1939),  evaluation  of  a  pi¬ 
lots’  emotional  adjustment,  temperament,  and  per¬ 
sonality  was  attempted. 

Both  psychological  and  medical  research  were  ini¬ 
tiated  during  the  First  World  War  when  many  pilots 
failed  to  complete  the  training  curriculum  and  the 
majority  of  casualties  were  reported  to  be  caused  by 
human  failure  rather  than  by  structural  failure  or 
combat.  The  allied  countries  stressed  different  aspects 
of  the  flying  task  while  developing  their  test  batteries. 
The  Italians  studied  perception  and  psychomotor 
activity  while  the  French  stressed  the  importance  of 
emotional  behavior  (Dockery  and  Isaacs,  1921).  The 
British  tests  were  concerned  primarily  with  physi¬ 
ological  parameters,  but  implied  certain  psychologi- 
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cal  correlations.  For  example,  volition  or  persistence 
was  judged  in  terms  of  the  candidate's  ability  to 
maintain  a  column  of  mercury  by  blowing  into  a 
manometer. 

According  to  McFarland  (1953)  the  limitations  of 
these  early  studies  in  aviation  psychology  included 
failures,  “1)  to  make  job  analyses  of  the  requirements 
for  satisfactory  performance,  2)  to  develop  methods  of 
rating  or  measuring  flight  performance  in  the  air  for 
correlation  with  tests  of  selection,  and  3)  to  determine 
how  well  the  tests  actually  differentiated  good  and 
poor  prospects  with  follow-up  studies  later  in  their 
flying  career.  Personal  judgments  rather  than  experi¬ 
mentally  determined  criteria  for  predicting  or  ap¬ 
praising  success  or  failure  characterized  the  early 
studies.”  (page  39) 

Pilot  Decision  Making  Research 

During  the  mid  1940s,  the  growth  of  aviation 
medicine,  and  the  need  for  quick  and  accurate  selec¬ 
tion  of  airmen  resulted  in  a  considerable  amount  of 
aviation  research.  Numerous  studies  referred  to  “deci¬ 
sion  making”,  “judgment”,  and  “pilot  error”  and 
implicated  the  lack  of  these  skills  in  aviation  accidents 
and  incidents.  Kelly  and  Ewart  (1942)  used  scales  on 
the  Purdue  Scale  for  Rating  Pilot  Competency  which 
included  both  achievement  and  the  intangible  factors 
of  judgment  and  emotional  stability.  A  sample  item  to 
assess  judgment  was  “how  good  is  his  judgment  in 
deciding  to  start  or  continue  a  flight  when  adverse 
factors  are  involved  such  as  weather?”  The  continuum 
of  answers  was  from  “extremely  cautious,”  “takes  no 
unnecessary  risks,”  “rarely  uses  poor  judgment,”  “takes 
some  unnecessary  risks,”  “takes  many  unnecessary 
risks,”  to  “extremely  reckless.”  Kalez  and  Hovde  (1953) 
reviewed  the  records  of  pilots  and  reported  on  what 
they  called  a  unique  psychological  group  of  pilots  who 
willfully  failed  to  use  checklists.  They  referred  to  this 
temporary  psychological  compulsion  as  an  error  in 
judgment.  The  pilots  who  were  involved  in  the  result¬ 
ing  accidents  presented  a  long  history  of  non-confor¬ 
mity  as  evidenced  by  their  flight  records.  Occasionally, 
they  were  below  average  aviators  in  every  measurable 
respect.  The  type  of  errors  in  which  these  investigators 
were  interested  were  classified  as  “pilot  errors”. 


The  nature  of  “pilot  error”  was  further  investigated 
by  Kunkle  (1946).  The  term  refers  to  all  the  defects 
which  a  pilot  may  exhibit  in  all  aspects  of  aviation, 
although  it  is  limited  in  that  it  primarily  refers  to 
accident  causing  behavior.  He  divided  the  topic  into 
primary  or  “non-emergency”  error  including  errors  in 
judgment  and  secondary  or  “emergency”  error  occur¬ 
ring  in  a  setting  of  tension  and  confusion  associated 
with  a  crisis  in  some  phase  of  flight.  Of  course,  the 
secondary  error  may  result  from  an  immediately  pre¬ 
ceding  pilot  error.  However,  true  emergencies  were 
responsible  for  only  a  small  minority  of  pilot  error 
accidents.  Kunkle  reports  that  World  War  II  pilot 
selection  tests  generally  failed  to  predict  those  who 
would  have  pilot  error  accidents  and  that  these  tests 
were  unable  to  assess  an  individual’s  judgment.  He 
further  implies  that  the  problems  involved  in  the 
operation  of  an  airplane  allow  full  scope  for  any  of  the 
various  manifestations  of  accident  proneness.  Re¬ 
viewing  the  accident  background  of  pilots  involved  in 
aircraft  accidents  revealed  a  previous  accident  pattern 
in  which  auto  accidents  were  indeed  conspicuously 
frequent.  He  concludes  that  there  is  a  direct,  but  by  no 
means  rigid,  relationship  between  past  performance 
on  the  ground  and  safety  record  in  the  air. 

Research  into  pilot  performance,  including  deci¬ 
sion  making,  continued  with  Henneman,  Hausman, 
and  Mitchell  (1947)  who  studied  Air  Force  pilot 
performance.  In  the  work  on  printed  classification 
tests  for  aircrew  members,  Guilford  and  Lacey  (1 947) 
report  that  of  all  the  traits  necessary  for  pilots,  judg¬ 
ment  stands  out  as  being  the  most  persistent  and 
universal.  However,  the  frequent  mention  of  judg¬ 
ment  for  the  pilot  presented  a  continuing  challenge  to 
break  it  down  into  manageable  components  and  de¬ 
vise  tests  for  it.  Guilford  and  Lacey  concluded  that  the 
judgment  factor  was  best  defined  by  a  work-planning 
type  of  item.  Furthermore,  items  calling  for  relatively 
complicated  estimates  involving  time,  as  well  as  dis¬ 
tance  and  size,  were  significantly  loaded  with  the 
judgment  factor  while  the  simpler  items  were  not.  The 
inference  which  may  be  drawn  is  that  the  judgment 
factor  is  a  thinking,  rather  than  a  perceptual  or  memory 
ability.  Guilford  and  Lacey  conclude  that  judgment 
was  highly  regarded  as  a  factor  and  received  consider¬ 
able  attention  (with  varying  degrees  of  success). 
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Recent  Aviator  Judgment  Evaluation  and 
Training  Research 

The  first  research  in  judgment  specifically  applied 
to  aviation  is  a  study  by  Thorpe,  Martin,  Edwards  and 
Eddows  (1976).  Situational  Emergency  Training 
(SET)  was  developed  for  the  US  Air  Force  in  response 
to  changing  the  number  of  crew  members  from  two 
(in  the  F-4  fighter)  to  one  (in  the  equally  complex  F- 
1 5) .  The  SET  program  was  intended  to  teach  the  pilot 
to  judge  the  relevant  dimensions  of  the  emergency, 
maintain  control  of  the  aircraft,  and  to  make  a  deci¬ 
sion  regarding  how  to  handle  the  problem.  This 
method  replaced  the  traditional  use  of  memorizing 
boldface  checklist  items  and  was  reported  to  be  well 
received  by  the  Air  Force. 

Jensen  and  Benel  (1977)  present  a  broad  outline  for 
a  judgment  training  and  evaluation  program.  They 
suggest  that  the  literature  outside  the  field  of  aviation 
should  be  used  to  develop  judgment  training  and 
evaluation  techniques  for  pilots.  Further,  they  add 
that  judgment  should  be  divided  into  intellectual  and 
motivational  aspects  to  establish  both  training  and 
assessment  approaches. 

Roscoe  (1980)  states  that  individuals  have  a  “pre¬ 
set,  though  modifiable,  decision  tendency  or  judg¬ 
ment  capability.”  Included  in  this  consideration  of  a 
“judgment  capability”  are  the  person’s  intellectual 
and  emotional  capabilities,  priorities,  self-esteem  and 
pride.  “The  pilot  who  has  been  trained  to  assess  flight 
alternatives  objectively  and  act  accordingly  in  all 
flight  situations  may  be  said  to  possess  good  flying 
judgment.”  If  this  is  the  case,  then  the  problem 
becomes  one  of  identifying,  measuring,  and  training 
the  skills  necessary  to  use  good  pilot  judgment. 

A  review  of  research  by  Giffin  and  Rockwell  (1984) 
shows  preliminary  work  in  designing  and  implement¬ 
ing  a  computer-assisted  testing  device  for  studying 
specific  types  of  pilot  decision  making.  Tests  of  a 
variety  of  candidate  hypotheses  concerning  the  style 
and  substance  of  pilot  resource  management  and 
decision  making  were  used  to: 

1.  Ascertain  the  role  of  pilot  background,  experi¬ 
ence,  and  knowledge  in  problem  diagnosis  and 
decision  making;  and 


2.  Describe  the  problem  solving  paths  in  sufficient 
detail  to  permit  the  ultimate  development  of 
various  models  of  pilot  behavior. 

Giffin  and  Rockwell  measured  pilot  response  to 
critical  in-flight  events  (CIFE)  using  a  computer  aided 
scenario  testing  system  (CAS)  in  a  test  of  forty- two 
subjects  with  varying  levels  of  flying  experience.  Sub¬ 
jects  first  entered  biographical  data,  then  took  a  knowl¬ 
edge  test  of  aircraft  systems  and  operations.  In  the 
diagnostic  scenario  procedure,  the  subject  was  told 
that  he  was  flying  a  Piper  Cherokee  Arrow  with 
information  on  equipment  and  performance  displayed. 
The  next  display  revealed  the  nature  of  the  mission 
and  symptoms  of  a  problem  being  encountered.  The 
subject  then  had  four  minutes  to  seek  information 
and  generate  a  diagnosis.  There  were  four  diagnostic 
scenarios.  When  the  time  was  up  or  a  diagnosis  had 
been  entered  by  touch  pane  entry,  the  time  history  of 
the  information  search  was  immediately  available  to 
the  experimenter.  Relevant  findings  in  the  study  in¬ 
cluded  (1)  knowledge  was  inversely  related  to  diag¬ 
nostic  inquiries,  i.e.,  knowledgeable  pilots  reached 
conclusions  (right  or  wrong)  more  rapidly  than  oth¬ 
ers;  (2)  less  experienced  pilots  tended  to  use  a  larger 
number  of  diagnostic  tracks  than  did  the  more  expe¬ 
rienced  pilots;  and  (3)  pilots  followed  a  wide  variety  of 
different  search  patterns  during  diagnosis. 

Berlin  et  ah,  (1982)  provided  the  initial  work  for 
the  writing  of  the  prototype  “Judgment  Training 
Manuals”  used  in  a  series  of  tests  by  the  FAA.  The 
manuals  were  designed  to  improve  judgmental  behav¬ 
iors  through  a  modified  behavioral  approach  applied 
through  instructor  pilots.  Five  “hazardous  thought 
patterns”  were  identified  and  applied  using  flight 
scenarios.  The  students  were  to  learn  how  to  identify 
these  hazardous  thought  patterns  and  to  replace  them 
with  more  rational  thinking. 

A  validation  study  using  the  “Judgment  Training 
Manual  for  Student  Pilots”  was  performed  by  Buch 
(Buch  and  Diehl,  1982)  on  Canadian  Air  Cadets.  A 
select  population  was  chosen  from  two  standardized 
flight  academies  in  Canada.  In  knowledge  tests  both 
control  and  experimental  groups  were  found  to  be 
adequate  and  equal  in  skills  and  knowledge.  The 
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results  of  the  experiment  show  that  the  experimental 
group  receiving  the  judgment  training  consistently 
made  “better”  decisions  than  the  control. 

Additional  judgment  training  manuals  have  been 
developed  for  specific  pilot  populations  in  civil  avia¬ 
tion.  Jensen  and  Adrion  (1984)  developed  a  program 
of  aeronautical  decision  making  for  instrument  pilots 
and  Jensen,  Adrion  and  Brooks  (1986)  developed  a 
manual  for  aeronautical  decision  making  for  commer¬ 
cial  pilots. 

Jensen,  Adrion  and  Maresh  (1986)  studied  the 
effectiveness  of  the  application  of  the  “DECIDE” 
model  to  aeronautical  decision  making.  The  “DE¬ 
CIDE”  model  of  decision  making  is  a  concept  of  the 
cognitive  judgment  process  consisting  of  six  elements 
arranged  in  a  closed  loop  system.  “DECIDE”  is  the 
acronym  for  Detect  change,  Estimate  significance  of 
change.  Choose  outcome  objectives.  Identify  plau¬ 
sible  action  options,  Do  best  option  and  Evaluate 
progress.  The  preliminary  results  in  this  evaluation 
indicated  the  model  has  great  potential  as  a  judgment 
training  tool  in  aviation.  The  “DECIDE”  trained 
subjects  demonstrated  clearer  thinking  patterns  in 
diagnosing  the  experimenter  induced  problems  dur¬ 
ing  a  simulated  flight  in  a  general  aviation  flight 
simulator.  They  also  showed  greater  concern  for  the 
safe  outcome  by  successfully  landing  the  “aircraft.” 

A  consistent  conclusion  in  the  majority  of  the 
aviation  decision  research  has  been  the  need  for  con¬ 
tinued  study  in  the  area.  Livak  (1983)  reviewed  cur¬ 
rent  literature  and  proposed  four  types  of  activities 
necessary  for  pilot  judgment  training.  First,  educa¬ 
tion  should  include  a  non-mandatoryjudgment  train¬ 
ing  program  for  all  pilots.  Second,  training  needs  to 
provide  specific  information  and  judgment  skills  re¬ 
quired  for  a  particular  license  or  rating.  Third,  certi¬ 
fication  requirements  should  assure  that  the  applicant 
possesses  and  can  demonstrate  sufficient  judgment. 
And,  fourth,  there  needs  to  be  a  rehabilitation  compo¬ 
nent  for  those  airmen  who  have  been  involved  in  an 
accident,  incident,  or  violation  in  which  the  investi¬ 
gating  official  felt  the  causal  factors  were  related  to 
poor  judgment. 

There  are  numerous  studies  that  relate,  at  least  in 
part,  to  the  subject  of  training  and  evaluating  pilot 
judgment.  Trollip  and  Ortony  (1977)  offered  insight 


into  real-time  simulation  in  computer  assisted  in¬ 
struction  that  is  applied  to  aviation.  A  simple  conver¬ 
sion  of  a  classroom  training  program  can  provide  a 
program  for  computer  assisted  instruction  (CAI). 

Personality  studies  by  Lester  and  Bombaci  (1984) 
and  Ashman  and  Telfer  (1983)  correlated  certain 
attributes  with  pilots.  These  studies  may  be  useful 
when  comparing  the  personality  profiles  of  pilots 
involved  in  accidents  and  incidents  with  selected 
profiles  of  the  general  population  to  determine  if  there 
are  any  particular  similarities  of  perhaps  neurotic  or 
psychotic  individuals.  To  date  this  determination  has 
not  been  made. 

Sociological  factors  have  also  been  considered  with 
respect  to  pilot  error  as  found  in  the  works  of  Urban 
(1983)  who  looked  at  the  urban/rural  differences  of 
pilots  involved  in  accidents.  The  universality  of  the 
problems  encountered  in  aviation  due  to  human  error 
is  evidenced  in  the  research  efforts  in  many  countries 
outside  of  the  United  States. 

Ground  (1984)  found  pilot-induced  factors  to  be 
responsible  for  69.3%  of  the  fatal  light  aircraft  acci¬ 
dents  in  the  United  Kingdom  from  1969-1981.  The 
position  was  taken  that  accidents  do  not  just  happen 
-  they  are  caused.  A  number  of  factors  were  included 
when  discussing  pilot  error  including;  forgetfulness, 
carelessness,  irresponsibility,  procrastination,  pride, 
ignorance  and  incompetence.  An  important  overall 
characteristic  of  judgment-caused  accidents  was  that 
these  accidents  seldom  had  a  single  cause. 

“Pilot  Error”  in  Accident  Investigation 

Due  to  imprecise  past  measurement  techniques  in 
accident  and  incident  causal  factors,  new  methodol¬ 
ogy  to  address  these  problems  has  recently  been  estab¬ 
lished  by  the  US  Air  Force.  Accident  investigation  can 
be  a  useful  adjunct  to  judgment  evaluation  training, 
testing,  and  tracking.  Therefore,  some  of  the  termi¬ 
nology  and  investigative  processes  used  in  the  new  Air 
Force  accident  and  incident  evaluations  are  discussed 
in  this  section.  This  information  is  incorporated  into  the 
framework  used  in  the  development  of  the  Jensen  Adrion 
Maresh  Judgment  Evaluation  Technique  QAMJET). 

The  dichotomy  between  training  and  accident  re¬ 
porting  with  respect  to  human  factors  in  the  Air  Force 
has  been  apparent  since  the  early  training  days.  The 
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cause  of  an  accident  may  be  classified  as  “human 
error”,  though  no  description  of  the  type  of  human 
error  is  given.  Was  it  failure  of  the  pilots*  perceptual 
system,  lack  of  knowledge,  physiological  failure,  or  a 
series  of  poor  decisions?  The  current  work  on  develop¬ 
ment  of  an  accident  reporting  system  congruent  with 
current  aeronautical  decision  making  training  pro¬ 
grams  will  add  greatly  to  the  understanding  of  the 
human  factors  problems.  It  is  this  understanding  of 
the  nature  of  problems  which  cause  accidents  that 
enhances  our  awareness  of  the  deficits  pilots  have  in 
these  abilities. 

New  guidelines  are  currently  being  established  for 
accident  investigation  team  members  which  include 
precise  definitions,  documentation  and  thorough 
evaluation  of  the  “psychological  concerns”  as  one  of 
the  areas  of  concentration.  A  workbook  has  been 
designed  to  aid  in  the  standardization  and  recording 
of  all  factors  involved  in  a  mishap  including  the 
human  factors  arena.  In  this  context,  a  mishap  is  an 
unplanned,  unintended  event  that  results  in  damage 
to  equipment  or  injury  to  personnel.  Mishaps  are 
broken  down  into  specific  categories,  antecedent 
events,  maneuver,  and  the  phases  of  flight. 

As  a  team  member  in  accident  investigations,  the 
human  factors/psychological  specialist  is  directed  to 
examine  human  factors  including  perception,  infor¬ 
mation  processing,  attention,  perceived  stresses,  fa¬ 
tigue,  coping  styles,  psychomotor  capabilities  and 
training.  A  distinction  is  made  between  “technical 
errors”  and  “judgmental  errors”  usually  involving 
higher-order  cognitive  processes.  Technical  errors  are 
objectively  inappropriate  individual  physical  or  men¬ 
tal  operations  such  as  missing  a  radio  call,  inaccurate 
altitude  or  airspeed  or  improper  switch  or  control 
operation.  Judgmental  errors  are  objectively  inappro¬ 
priate  selection  of  a  course  of  action  constituted  by  a 
number  of  subsequent  subtasks.  Examples  include 
making  an  approach  to  below  minimums  or  accepting 
an  aircraft  or  personal  condition  inappropriate  to 
anticipated  mission  demands. 

Types  of  errors.  Morris  and  Rouse  (1985)  differen¬ 
tiate  between  “slips”  and  “mistakes”  as  types  of  human 
errors.  Slips  are  considered  errors  of  action  occurring 
during  a  well  trained  activity.  The  slip  is  usually 
brought  on  by  a  distraction  or  a  preoccupation.  Since 


these  errors  occur  in  well  established  routines,  they  are 
usually  unmonitored.  When  they  do  occur  and  are 
subsequently  discovered,  they  add  to  the  distraction. 
Specific  types  of  slips  include  habit  pattern  interfer¬ 
ence,  perceptual  set,  and  omission  or  repetition  of 
steps  in  a  sequence  (e.g. ,  checklist  items  being  skipped, 
or  repeated.) 

A  “mistake”  type  of  error  concerns  judgment  and 
decision  making  issues.  Judgmental  errors  are  likely 
when  more  than  two  or  three  variables  must  be  simul¬ 
taneously  considered.  They  may  also  occur  when  an 
inappropriate  solution  has  been  successful  in  past 
similar  situations  and  when  solution  options  are  novel. 
Poor  judgment  is  considered  to  be  the  failure  to 
realistically  assess  the  significance  and  priority  of 
information  from  the  environment.  Assuming  ad¬ 
equate  quality  and  quantity  of  information,  the  poor 
judgment  is  due  to  an  anomaly  of  attention  or  anomaly 
of  motivation. 

Attention  and  Motivation.  Anomalies  of  attention 
are  the  misallocation  or  untimely  interruption  of 
attention  to  a  task  easily  influenced  by  fatigue  or  other 
stressors.  The  types  of  anomalies  of  attention  include 
channeled  attention,  cognitive  saturation,  distrac¬ 
tion,  fascination,  inattention  and  habit  pattern  inter¬ 
ference  or  substitution.  Anomalies  of  motivation  are 
characteristics  of  a  person’s  value  system  which  may 
result  in  unsafe  acts.  Types  of  motivation  anomalies 
include  excessive  motivation,  under  motivation  and 
misplaced  motivation. 

Workload  and  Stress.  Workload  is  considered  for 
investigative  purposes  as  contributing  to  errors.  Per¬ 
ception  of  the  amount  of  stimulation  as  too  little  or 
too  much  will  have  an  effect  on  the  number  of  errors. 
On  the  other  hand,  Hart  and  Bortolussi  (1984)  con¬ 
sidered  pilot  errors  as  a  source  of  workload  rather  than 
a  symptom.  This  viewpoint  may  be  advantageous 
when  reconstructing  the  sequence  of  events  in  a  mis¬ 
hap.  Thus,  the  occurrence  of  mistakes  and  slips  add  to 
the  mental  workload. 

Another  view  of  the  mental  workload  involved  in 
the  specifically  active  and  adaptive  responses  to  the 
demands  posed  by  a  complex  task  is  offered  by  Wiener 
(1985).  To  identify  the  operational  components  of 
the  central  mechanisms  involved  in  information  han¬ 
dling  and  decision  making,  the  subsystem  components 
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were  distinguished.  This  enables  a  measure  of  the 
stress  undergone  by  each  subsystem.  “Stress”,  in  psy¬ 
chological  terms  means  the  active  responses  which 
are  specifically  adaptive  or  adjusted  to  the  external 
stress  which  maintains  homeostasis.  Psychological 
homeostasis  is  an  ability  to  “cope”  however  intense 
the  performance  level  that  is  its  cost  or  stress,  “Strain” 
is  evident  in  the  non-maintenance  of  homeostatic 
levels.  In  other  words,  under  strain  the  pilot  is  asking 
to  be  overstressed.  Psychological  strain  is  shown  by  the 
inability  to  “cope”  with  the  imposed  load  or  stressor. 

The  breakdown  of  the  system  is  as  follows: 

1 .  The  discrimination  subsystem  is  where  incoming 
signals  are  recognized  and  coded.  The  demand 
(stress)  placed  on  this  coding  system  could  be 
quantified  in  terms  of  the  number  of  identifica¬ 
tions  per  unit  time. 

2.  The  decision  making  (or  choice)  subsystem  con¬ 
nects  the  identification  subsystem  to  the  output 
subsystem.  The  rate  at  which  connections  are 
made  for  passing  on  the  coded  input  signals  is  the 
measure  of  the  response  or  the  stress  placed  on 
this  subsystem. 

3.  The  output  subsystem  is  the  number  of  central 
output  signals  per  unit  time  sent  to  the  executive 
muscles  for  the  performance  of  the  task.  The  rate 
of  output  is  a  measure  of  the  response  or  stress 
placed  on  the  system. 

4.  The  corrective  feedback  subsystem  regulates  the 
accuracy  of  the  output.  The  operation  is  assessed 
in  terms  of  errors  made  in  performing  a  set  task, 
but  error  evaluation  is  more  properly  regarded  as 
a  measure  of  strain. 

Generally,  the  measure  of  response  to  a  mental  task 
is  restricted  to  only  the  output  system  as  a  measure  of 
stress.  A  measure  of  strain  could  be  taken  at  the 
corrective  subsystem.  One  special  category  of  internal 
factors  needs  also  to  be  considered  in  this  general 
model.  These  host  factors  include  the  phases  of  the 
circadian  rhythm,  time  period  (pre-  or  post-weekend, 
holidays),  quality  of  sleep,  repetitiveness  of  the  task, 
change  of  shift,  and  relationship  to  meals.  Drug  and 


alcohol  use  can  be  included  here  as  well.  These  factors 
influence  the  psychological  response  and  add  to  or 
modify  the  internal  stress  system. 

Jensen  Adrion  Maresh  Judgment  Evaluation 
Technique  QAMJET) 

Figure  1  is  a  detailed  model  that  describes  a  series 
of  eight  steps  in  the  decision  making  process.  This 
model  is  an  adaptation  of  an  eight-step  judgment 
model  first  offered  in  a  report  for  the  US  Air  Force 
0ensen,  Adrion,  and  Maresh,  1987)  and  later  pre¬ 
sented  in  Jensen  (1995).  This  model  is  specifically 
designed  to  describe  the  aviation  cognitive  judgment 
process.  Understanding  each  step  and  how  the  steps 
are  interrelated  is  important  to  the  development  of 
better  prescriptive  models  needed  for  intervention 
development.  The  complete  description  of  this  pro¬ 
cess  as  offered  in  Jensen  (1995)  is  not  necessary  here. 
For  our  purposes,  it  presents  a  picture  of  our  under¬ 
standing  of  the  information  processing  aspect  of  avia¬ 
tor  decision  making  that  must  be  examined  in  this 
program.  The  blocks  indicate  the  steps  in  the  process 
and  the  descriptors  on  the  right  indicate  mental  fac¬ 
tors  that  enter  into  each  step. 

A  more  practical  model  of  pilot  judgment  was 
presented  in  the  Jensen  and  Benel  (1977)  report.  This 
model  had  two  parts:  1)  an  ability  and  2)  a  motivation 
as  shown  below: 

Part  I:  Rational  Judgment: 

The  ability  to  discover  and  establish  the  relevance  of 
all  available  information  relating  to  problems  of  flight, 
to  diagnose  these  problems,  to  specify  alternative  courses 
of  action,  and  to  assess  the  risk  associated  with  each 
alternative. 

Part  II:  Motivational  Judgment: 

The  motivation  to  choose  and  execute  a  suitable  course 
of  action  within  the  available  time  frame. 

Where: 

a.  The  choice  could  be  either  action  or  no 
action  and, 

b.  “Suitable”  is  a  choice  consistent  with  “soci¬ 
etal”  norms. 
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Sensation:  sights,  sounds,  smells,  g-forces,  muscle  tension 


Perception:  expectancy,  information  processing,  short-term 
memory 


Cognition:  knowledge,  information  processing,  long-term 
memory 


Recollection:  inquiry,  com.munication, 
long-term  memory 


Calcuiation:  problem  solving,  probability 
assessment 


Motivation:  risk-taking  attitudes,  non¬ 
flight  pressure,  commitment,  economics, 
stress,  physiological  factors 


Management:  command  authority,  time  considerations 


Decision  implementation:  psycho-motor  skill,  control 
manipulation 


Figure  1.  Detailed  judgment  model  (Jensen,  1995) 
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Rational  Judgment.  The  first  part  of  good  pilot 
judgment  is  the  mental  ability  of  the  pilot  to  detect, 
recognize,  and  diagnose  problems,  to  establish  avail¬ 
able  alternatives  and  to  determine  the  risk  associated 
with  each  alternative.  This  part  is  purely  rational,  and 
if  it  could  be  used  alone  (which  is  not  possible),  would 
allow  problem  solving  using  mathematical  functions 
in  much  the  same  manner  as  a  computer.  This  does 
not  mean  it  would  be  error  free;  it  uses  information 
that  is  probabilistic  and  therefore,  predicts  outcomes 
that  are  not  certain.  In  addition,  rational  judgment 
depends  upon  the  amount,  type,  and  accuracy  of  the 
information  stored  in  the  pilot’s  memory  as  well  as  his 
or  her  learned  capabilities  to  retrieve  and  process 
information.  To  optimize  rational  judgment  requires 
high  levels  of  knowledge,  experience,  organized  men¬ 
tal  structures,  and  systematic  computational  and  prob¬ 
lem  solving  abilities. 

Motivational  Judgment.  The  second  part  is  the 
motivational  or  bias  aspect  of  judgment.  The  empha¬ 
sis  is  on  the  directional,  rather  than  the  aspects  of 
motivation  dealing  with  intensity.  This  part  of  judg¬ 
ment  says  that  humans  (and  pilots)  base  their  deci¬ 
sions,  in  part,  upon  bias  factors  or  tendencies  to  use 
less  than  purely  rational  (as  defined  by  society)  infor¬ 
mation.  These  factors  include  immediate  gratifica¬ 
tion  such  as  ego,  adventure,  commitment,  duty,  social 
pressure,  and  emotional  arousal  in  the  form  of  worry, 
fear,  stress,  anxiety,  and  euphoria,  as  well  as  more  long 
term  biases  such  as  risk-taking  attitudes,  and  person¬ 
ality  factors  (e.g.,  fear  of  failure  and  defensiveness). 
Optimizing  motivational  judgment  requires  both  1) 
an  awareness  of  biasing  factors  and  2)  a  will  (motiva¬ 
tion)  to  suppress  these  error  producing  factors  so  that 
decisions  can  be  made  on  the  basis  of  relevant  safety 
factors  from  the  physical  world. 

At  this  time,  the  rational  aspect  of  pilot  judgment 
has  received  very  little  attention.  However,  there  is 
much  in  the  literature  outside  of  aviation  including 
stock  brokers,  livestock  judges,  and  medical  diagnos¬ 
ticians,  indicating  that  this  aspect  of  judgment  can  be 
taught.  In  each  of  the  areas  studied,  judgmental  train¬ 
ing  occurs  over  a  fairly  long  apprenticeship  program 
in  which  the  trainee  observes  the  expert  making 


decisions  and  learns  by  these  observation.  Rouse  and 
his  colleagues  have  performed  a  series  of  experiments 
to  develop  fault  diagnosis  training  systems  to  be 
administered  on  computer  (Rouse,  1979).  As  men¬ 
tioned  earlier,  one  demonstration  study  at  The  Ohio 
State  University  (Jensen,  Adrion,  and  Maresh,  1986) 
has  shown  the  effectiveness  of  the  DECIDE  model  in 
teaching  rational  judgment  to  pilots. 

On  the  other  hand,  the  motivational  aspect  of  pilot 
judgment  has  received  the  bulk  of  research.  Early 
efforts  following  the  Jensen  and  Benel  study  focusing 
on  this  part  of  the  model  have  shown  that  motiva¬ 
tional  training  can  be  effective.  The  model  used  in  all 
of  these  studies  may  be  called  the  attitude  model  or 
five  hazardous  attitudes:  Anti-Authority,  Impulsivity, 
Invulnerability,  Macho,  and  Resignation.  An  aware¬ 
ness  of  these  attitudes,  that  are  found  to  some  extent 
in  everyone,  can  help  to  develop  a  more  positive  and 
rational  approach  toward  flying  decisions.  Training 
studies  using  this  model  have  demonstrated  that  pilot 
decision  making  improves  anywhere  from  13%  to 
1 00%  as  a  result  of  attitude  training  (Buch  and  D  iehl, 
1982;  Telfer,  1987;  1989;  Diehl  and  Lester,  1987; 
Diehl,  1992;  Fox,  1991;  Alkov,  1991).  Impressive 
results  have  also  been  reported  in  two  helicopter 
operational  training  studies.  Petroleum  Helicopter 
Inc.  (PHI)  and  Bell  Helicopter  have  both  offered  the 
attitude  method  of  judgment  training  to  large  num¬ 
bers  of  helicopter  pilots.  PHI  has  reported  a  54% 
reduction  in  accidents  after  giving  this  training  to 
their  pilots.  In  two  studies.  Bell  Helicopter  reported  a 
36%  decrease  and  a  48%  decrease  in  accident  rates 
after  the  training.  Both  organizations  point  to  the 
judgment  training  as  the  most  important  tool  now 
available  to  improve  safety  in  helicopter  flying. 

However,  attitude  training  as  it  is  formulated  in  the 
original  Embry-Riddle  Aeronautical  University  manu¬ 
als  (Berlin  et  ah,  1982)  has  negative  connotations  and 
its  benefits  seem  to  have  reached  a  plateau.  Alternative 
approaches  are  needed  which  focus  on  the  other  half 
of  the  Jensen  and  Benel  model  emphasizing  informa¬ 
tion  processing.  This  new  emphasis  requires  funda¬ 
mental  research  into  human  decision  making  and 
modeling. 
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Expertise  Research 

A  prerequisite  to  this  project  was  a  definition  of  the 
“expert”  pilot.  The  current  literature  on  expertise  in 
aviation  is  rather  limited;  therefore,  a  review  of  the 
research  on  expertise  in  other  domains  was  conducted 
as  well.  The  full  range  of  methods  of  analysis  in 
cognitive  psychology  and  cognitive  engineering  can 
be  applied  in  the  examination  of  phenomena  associ¬ 
ated  with  a  particular  type  of  expertise.  These  include 
performance  analysis,  expert-novice  comparisons,  and 
extensive  studies  of  single  subjects.  The  identification 
of  a  collection  of  tasks  that  can  capture  superior 
performance  is  often  not  easy.  This  study  used  a  large 
number  of  tasks  for  the  subjects  studied. 

Experts  engage  in  a  number  of  complex  mental 
activities,  involving  the  ability  to  plan  and  reason, 
that  rely  on  mental  models  and  internal  representa¬ 
tions.  Charness  (1981)  found  that  the  depth  to  which 
a  possible  move  sequence  for  a  chess  position  was 
explored  was  closely  related  to  the  level  of  chess  skill. 
Charness  (1989)  found  that  expertise  at  the  game  of 
bridge  was  closely  linked  with  the  capacity  to  generate 
successful  plans  for  playing  the  cards  in  the  optimum 
order.  Similarly,  in  medical  diagnosis,  many  different 
pieces  of  information  from  different  sources  must  be 
integrated.  Since  this  information  is  not  available  at 
the  same  time,  the  internal  representation  of  the 
presented  medical  information  must  be  sufficiently 
precise  to  allow  extensive  reasoning  and  evaluation  of 
consistency,  but  also  must  be  sufficiently  flexible  to 
allow  reinterpretation  as  new  information  becomes 
available  (Lesgold  et  ah,  1985). 

Sloboda  (1991)  made  the  distinction  between  “rou¬ 
tine”  and  “adaptive”  expertise.  Routine  expertise  is 
the  reliable  attainment  of  specific  goals  within  a 
specific  domain.  Whereas  routine  experts  are  able  to 
solve  familiar  types  of  problems  quickly  and  accu¬ 
rately,  they  have  only  modest  capabilities  in  dealing 
with  novel  types  of  problems.  Adaptive  experts  can  make 
an  appropriate  response  to  a  situation  that  contains  a 
degree  of  unpredictability  and  may  be  able  to  invent  new 
procedures  derived  from  their  expert  knowledge. 

Expertise  theory  has  been  evolving  since  the  late 
1950's  with  as  many  theories  proposed  as  there  are 
theorists.  Early  research  on  decision  making  stressed 


mathematical  models  that  either  measured  decision 
biases  or  used  game  theory  and  economics.  Tradi¬ 
tional  research  evolved  from  two  paradigms,  formal¬ 
ist-empiricist  and  rationalist.  Formalist-empiricist 
research  focused  on  behavioral  testing  of  formal  mod¬ 
els  and  neglected  the  cognitive  processes  underlying 
the  subjects’  decisions.  The  rationalist  approach  es¬ 
tablishes  a  formal  view  of  normative  standards  and 
accounts  for  reasoning  in  terms  of  a  set  of  unrelated 
cognitive  mechanisms.  Experimentally  this  approach 
demonstrates  the  errors  that  take  place  between  che 
normative  model  and  the  actual  decisions. 

In  an  effort  to  make  the  decision  research  useful  in 
context,  the  Army  Research  Institute  Office  of  Basic 
Research  (Johnson,  Rouse  and  Rouse,  1980),  began 
investigating  planning,  problem  solving,  and  decision 
making  in  naturalistic  settings.  These  research  meth¬ 
odologies  focus  on  decision  processes  taking  place  in 
realistic,  dynamic,  and  complex  environments  and 
their  real-world  outcomes.  The  naturalistic  paradigm 
investigates  the  way  people  actually  respond  to  com¬ 
plex  environments  targeting  the  functions  that  cogni¬ 
tive  processes  serve.  Whereas  decision  biases  in  the 
traditionalists’  viewpoint  are  violations  of  consistency 
constraints  imposed  by  the  theory  or  norm,  the  natu¬ 
ralists  view  decision  errors  in  the  real  settings  differ¬ 
ently.  The  naturalistic  approach  emphasizes  cognitive 
representations  and  processes,  but  as  Cohen  (1993) 
explains  “evaluation  of  reasoning  is  more  subtle  and 
demanding:  no  longer  a  cookie-cutter  comparison 
between  performance  and  an  unquestioned  norma¬ 
tive  template.  In  the  naturalistic  framework,  the  reci¬ 
procity  between  normative  and  descriptive  concerns 
that  characterized  the  formal-empiricist  approach  can 
be  retained. ..if  cognitive  as  well  as  behavioral  criteria 
are  incorporated  into  normative  modeling.”  (Page  50). 

Moving  from  the  theoretical  to  the  applied  re¬ 
search,  Adams  (1993),  in  a  summary  of  the  expertise 
literature  offers  seven  basic  characteristics  of  what  he 
terms  “Expert  Decision  Making”  which  are  relevant 
to  experts  in  all  domains.  These  are: 

1 .  Superior  memory.  Expert’s  recall  seems  to  exceed 
the  limits  of  short-term  memory.  Autonomous  in¬ 
formation  processing  frees  up  greater  storage.  Experts 
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excel  in  long-term  memory  as  well.  Elaborations, 
associations  and  inferences  expedite  recall. 

2.  Goal  oriented.  The  knowledge  of  experts  is  highly 
goal  oriented.  Concepts  are  bound  to  procedures 
for  their  applications  and  to  conditions  under 
which  these  procedures  are  useful.  This  func¬ 
tional  knowledge  is  strongly  related  to  task  de¬ 
mands  and  goals  invaluable  in  problem  solving. 

3.  Fast  access.  Experts  can  solve  problems  more 
quickly  and  accurately  than  non-experts.  They 
are  faster  at  skill-based  tasks,  these  tasks  require 
less  attention  capacity,  and  they  have  more  time 
to  do  cognitive  tasks.  They  arrive  at  solutions 
without  conducting  an  extensive  search  of 
memory. 

4.  Opportunistic  planning.  Experts  can  revise  pro¬ 
duction  rules  and  simultaneously  access  multiple 
interpretations  of  a  situation. 

5.  Adaptive.  There  are  both  routine  and  adaptive 
experts.  Both  have  outstanding  accuracy,  speed 
and  automaticity  in  decision  making  and  prob¬ 
lem  solving.  Routine  experts  are  more  limited  in 
their  capability  to  perform  new  or  ill-structured 
problems.  Adaptive  experts  have  the  ability  to 
creatively  respond  to  ill-structured  or  ambiguous 
new  problems  with  a  considerable  chance  to  find 
a  successful  outcome. 

6.  Self-monitoring.  Greater  knowledge  bases  and 
different  knowledge  representations  of  experts 
allow  more  time  to  predict  problem  difficulty  on 
the  basis  of  underlying  principles  and  to  monitor 
how  they  should  allocate  time  and  resources  to 
solving  the  problem.  Experts  are  more  sensitive 
to  informational  feedback  useful  to  problem  so¬ 
lution. 

7.  Perceptual  superiority.  Experts  have  the  ability 
to  perceive  meaningful  patterns  using  the  organi¬ 
zation  of  their  knowledge  base.  Pattern  recogni¬ 
tion  occurs  so  rapidly  that  it  takes  on  the 
characteristics  of  instantaneous  insight  or  intuition. 

In  his  effort  to  develop  a  personality  instrument  to 
profile  “safe”  and  “unsafe”  pilots,  Rodgers  (1994) 
developed  another  model  of  the  expert  pilot.  In  his 
model  an  expert  pilot  is  an  individual  who  allows  for 
few  errors,  possesses  almost  total  objectivity,  excellent 


perceptiveness,  and  a  tolerance  for  risk  without  any 
tendency  to  pursue  risk  for  its  own  sake.  The  expert 
pilot  must  maintain  the  ability  to  compartmentalize 
and  prioritize  demands  throughout  the  spectrum  of 
situations  from  boredom  (repetition)  through  total 
mayhem  (the  unexpected,  unknown  emergency). 

The  current  study,  though,  addressed  from  the 
viewpoint  of  expertise  literature,  the  question,  “What 
does  an  expert  pilot  do  (or  not  do)  that  an  average 
pilot  does  or  does  not  do?”  Building  upon  the  previ¬ 
ous  research,  this  study  sought  to  develop  a  better, 
operational  definition  of  an  expert  pilot,  to  delineate 
the  characteristics  of  expert  pilots  that  differentiate 
them  from  novices,  and  to  examine  the  degree  to 
which  pilots  who  might  be  regarded  as  experts  possess 
those  characteristics.  To  do  this,  we  accomplished  a 
series  of  studies  moving  from  general,  loosely  struc¬ 
tured  exploratory  studies  to  more  tightly  controlled 
experimental  procedures. 

The  project  was  composed  of  four  studies.  During 
the  first  study,  pilots  were  interviewed  to  conceptual¬ 
ize  a  working  definition  of  the  “expert”  pilot.  The 
second  study  consisted  of  administering  a  structured 
interview  to  thirty  pilots  who  met  the  criteria  of 
“expert”  pilot  from  the  initial  unstructured  inter¬ 
views.  The  third  study  of  the  project  evaluated  the 
candidate  definition  of  “expert”  pilot  in  three  types  of 
aircraft  making  up  the  mid-altitude  flying  domain: 
Beech  P-Baron,  Piper  Malibu,  and  Cessna  P-210. 
National  Transportation  Safety  Board  accident  re¬ 
ports  were  reviewed  for  these  three  types  of  aircraft 
and  an  experimental  flight  scenario  was  developed.  In 
study  four,  the  entire  scenario  was  read  to  subjects  and 
their  responses  were  recorded.  The  recordings  were 
transcribed  and  encoded  into  specific  categories  for 
data  analysis.  All  four  of  these  studies  are  discussed  in 
detail  in  the  following  sections. 

STUDY  1 

SEMI-STRUCTURED  INTERVIEWS 

The  first  study  consisted  of  a  set  of  semi-structured 
interviews  to  begin  the  quest  for  an  understanding  of 
expertise  in  aviation  and  to  determine  workable  ter¬ 
minology  to  be  used  with  aviators  in  discussing  exper¬ 
tise.  From  these  interviews,  three  levels  of  pilot 
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performance  emerged  which  were  given  the  designa¬ 
tions  of ‘"below-average”,  ‘‘average”,  and  “above-aver¬ 
age”.  To  date,  this  terminology  has  proved  useful  in 
discussions  with  pilots  concerning  expertise.  These 
interviews  also  investigated  possible  representative 
tasks  for  use  in  evaluating  above-average  performance 
in  aviation  by  eliciting  critical  event  scenarios  from 
the  subjects. 

Subjects 

The  initial  semi-structured  interviews  were  admin¬ 
istered  to  1 0  highly  experienced  pilots  with  an  average 
number  of  years  flying  of  32.5,  and  average  total  flight 
time  of  approximately  13,500  hours.  All  of  the 
interviewees  were  flight  instructors  and  six  were  also 
designated  pilot  examiners.  These  same  individuals 
agreed  to  participate  in  future  phases  of  the  project. 

Procedure 

The  initial  verbal  protocol  analysis  for  aviators 
began  with  the  questionnaire  shown  in  Appendix  A. 
This  instrument  was  designed  to  elicit  responses  from 
the  subjects  to  aid  in  the  establishment  of  working 
definitions  used  in  this  project  (e.g.  “expert  pilot”, 
“novice  pilot”) .  The  questionnaire  was  given  orally  to 
each  individual  pilot  as  a  part  of  the  interview  and 
their  responses  were  recorded  by  the  experimenter. 
The  questions  were  open  ended  and  participants  were 
encouraged  to  expand  and  elaborate  on  their  answers. 

Results 

The  following  is  a  summary  of  the  ideas  provided 
by  the  10  pilots  in  the  preliminary  interview  orga¬ 
nized  along  the  lines  of  the  topics  requested: 

1 .  How  can  you  distinguish  how  “good”  a  pilot  is? 
''How  do  you  know  you  are  in  love?"' 

"Good  pilots  are  a  part  of  the  machine.  ” 

"They  have  a  feel  for  it.  ” 

"A  good  pilot  can  put  it  all  together,  they  have  the 
rote  knowledge,  understanding,  and  ability  to  apply 
this.  ” 

"A  good  pilot  is  smooth,  doesn  "tjerk,  can  anticipate 
what  he  is  going  to  do  and  is  methodical" 

"A  good  pilot  coordinates  well.  " 

"Confidence  and  smoothness. " 


2.  If  you  were  to  group  pilots  into  three  categories 
based  on  how  “good”  they  are,  what  would  you 
name  the  categories? 

C2itc^ory  A.  Ace,  professional,  natural,  good,  above 
average,  accomplished,  best,  unique. 

Category  B.  Run  of  the  mill,  mimic,  competent, 
average,  comfortable. 

Category  C.  Worst,  poor,  amateurs,  dangerous, 
rote,  below  average. 

3.  What  did  you  base  your  categories  on? 
Categories  were  established  primarily  by  answer¬ 
ing  the  first  question.  Ability  to  explain,  under¬ 
stand  his  machine,  and  the  ability  to  verbalize. 
"What  I  have  seen.  ” 

4.  Describe  a  “Category  A”  pilot.  Can  you  name 
three? 

"Confident. " 

"Good  stick  and  rudder,  and  can  fly  big  and  little 
airplanes. " 

"Has  the  "magic"  which  is  imagination  with  respect 
and  awe. " 

"Always  a  student  of  aviation.  " 

"Has  a  strong  desire  to  fly.  " 

"Part  of  the  aircraft.  " 

5.  Describe  a  “Category  B”  pilot.  Can  you  name 
three? 

"Proficient  stick  and  rudder,  but  unaware.  " 

"Safe  enough  to  get  there  and  back.  " 

"Not  really  with  the  aircraft.  " 

6.  Describe  a  “Category  C”  pilot.  Can  you  name 
three? 

"Poor  attitude  control. " 

"Mechanical pilot. " 

"Can't put  mechanical  skills  together  with  what's 
going  on." 

"Dangerous. " 

"Doesn't  keep  up  with  anything.  " 

"Narrow  field  of  inquisitiveness.  " 

7.  What  is  the  best  maneuver  to  use  to  determine 
how  “good”  a  pilot  is? 
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The  maneuvers  mentioned  were  lazy  eights,  once 
around  the  patch,  the  FAA  weave,  ILS,  minimum 
controllable  airspeed,  and  steep  turns, 

8.  Name  the  three  most  memorable  events  you  have 
had  as  a  crewmember  in  any  aircraft. 

This  question  was  asked  to  begin  a  collection  of  “critical 
incidents**  and  were  gathered  for  future  research  use. 
The  responses  were  both  positive  (enjoyable)  events  and 
negative  (emergencies  encountered), 

9.  Would  you  be  willing  to  participate  in  future 
phases  of  this  project? 

All  initial  participants  were  willing  to  continue 
participation , 

Discussion 

From  these  responses  several  candidate  definitions 
of  the  levels  of  pilot  expertise  emerged.  It  was  clear 
from  discussion  with  these  very  high-time  instructor 
and  examiner  pilots  that  there  is  a  need  to  differentiate 
between  types  of  skill  needed  to  be  an  “expert”  aviator 
(i.e.,  mechanical  skills,  knowledge,  and  judgment). 
One  theme  that  was  repeated  frequently  was  that  “farm 
boys  often  make  above  average  pilots,  but  just  because  he 
can  drive  the  tractor  doesn’t  make  him  an  astronaut.” 
From  our  analysis  of  the  semi-structured  interview  re¬ 
sults,  the  following  definitions  were  developed: 

STUDY  1 

INITIAL  DEFINITIONS  OF  THE 
ABOVE  AVERAGE  PILOT 

Skills: 

1.  Mechanical  ability  -  either  natural  or  learned, 
(innate  vs.  trained) 

2.  Reasoning  -  assimilate  information,  process  in¬ 
formation,  and  perform 

Procedural  Knowledge: 

1 .  Aircraft  systems  (must  know  pilot  operating  hand¬ 
book) 

2.  Rules  (regulations,  operations  specifications) 

3.  Procedures  for  maneuver 

4.  Anticipation  (Situational  Awareness) 

5.  Strategies  and  planning 


Learning  and  Performance  Strategies: 

1.  Desire  to  excel  and  perform  in  aviation 

2.  Focus  and  ability  to  focus 

3.  Situational  awareness  within  the  time  frame 

4.  Decision  making 

5.  Practice  (even  a  “natural  has  to  practice”) 

6.  Goal  to  be  the  best  possible  and  always  working 
toward  that  goal 

7.  Keep  expanding  goals  -  always  a  student  of  aviation 

Confidence: 

1 .  Good  ego  strength  (“I  can  do  it”) 

2.  Not  over  confident 

3.  Dedication  to  task 

4.  Confident  enough  to  admit  an  error 

Motivation: 

1.  Ego  factor  -  drives  them  to  be  the  best 

2.  Do  it  for  yourself 

3.  Enjoys  the  challenge  of  flying 

4.  Perceived  money  and  prestige 

5.  Wants  freedom  and  control 

6.  Has  the  “X”  factor  (Indescribable  “magic”) 

In  summary,  the  initial  results  of  semi-structured 
interviews  provided  suggestions  for  terminology  to  be 
used  in  further  interviews  and  in  verbal  protocol 
simulations.  After  much  discussion  and  review  of 
other  studies  (Ericsson  and  Smith,  1991;  Dreyfus  and 
Dreyfus,  1986),  we  decided  to  use  “above  average,” 
“average,”  and  “below  average”  as  the  three  classifica¬ 
tions  of  pilots  while  working  with  the  subject-pilots  in 
subsequent  studies.  Our  abbreviated  definitions  of 
these  terms  were: 

Above  Average:  Highly  motivated,  confident  (but 
not  overconfident),  superior  learning  and  performance 
skills,  has  the  “magic,  ** 

Average:  Adequate  learning  and performance  skills  to 
complete  the  task. 

Below  Average:  Lacking  in  motivation  and! or  the 
ability  to  learn  andlor perform  consistently. 


STUDY  2 

STRUCTURED  INTERVIEWS 

The  second  study  consisted  of  structured  inter¬ 
views  with  a  different  set  of  pilots  who  were  engaged 
in  instructing  and/or  evaluating  pilots.  All  had  expe¬ 
rience  in  the  three  aircraft  making  up  the  domain.  The 
purpose  of  these  interviews  was  to  investigate  the 
cognitive  processes  and  learning  strategies  of  the  above 
average  pilot  in  order  to  refine  our  working  definition 
of  aviator  expertise  and  develop  distinguishing  char¬ 
acteristics  of  the  expert  in  this  domain. 

Procedure 

Thirty  highly  experienced  pilots  (average  flight 
time  of  over  5,000  hours)  were  interviewed  using  the 
Structured  Interview  form  shown  in  Appendix  B. 
Subject  responses  were  recorded  by  the  experimenter. 
The  interview  format  was  again  open-ended  as  in 
Study  1.  Questions  were  added  to  investigate  the 
motivational  aspect  of  pilot  continuous  learning,  which 
subjects  in  Study  1  had  indicated  was  an  important 
factor  in  above  average  pilots.  The  responses  were 
compiled  and  analyzed. 

Results  and  Discussion 

The  responses  to  the  structured  interviews  were 
extensively  reviewed  for  repetitive  answers  and  state¬ 
ments  across  the  participants.  A  strong  theme  and 
pattern  to  the  interviews  emerged  leading  to  a  prelimi¬ 
nary  definition  and  a  set  of  distinguishing  character¬ 
istics  of  the  expert  pilot  in  this  domain.  The  preliminary 
definition  of  an  above-average  or  expert  pilot  is  one 
who  is  highly  motivated,  confident  (but  not  overcon¬ 
fident),  has  superior  learning  and  performance  skills, 
applies  those  skills  in  a  changing  environment,  and 
possesses  a  type  of  judgment  described  by  many  as 
“magic”  or  “natural.”  In  pilot  language  the  expert 
pilot  becomes  “part  of  the  machine  and  flows  within 
the  ‘flying  space’.”  The  pilot  expert  in  this  domain  is 
one  who: 

1.  Possesses  self-confidence  in  his  or  her  skills  as  a 
pilot, 


2.  Is  highly  motivated  to  learn  all  there  is  to  know 
about  the  flight  domain  and  practices  their  skills 
constantly, 

3.  Has  superior  ability  to  focus  on  the  necessary  task 
and  change  that  focus  at  the  slightest  hint  that  a 
change  is  needed, 

4.  Has  excellent  situational  awareness  (flight  envi¬ 
ronment,  location  of  other  aircraft,  terrain,  navi¬ 
gation,  communications,  weather,  etc.), 

5.  Is  highly  cognizant  of  the  machine  including 
noise,  vibration,  and  engine  indications, 

6.  Is  always  vigilant  for  the  unusual,  abnormal,  or 
emergency,  and  mentally  makes  contingency 
plans, 

7.  Has  superior  mental  capacity  for  problem  diag¬ 
nosis,  risk  assessment,  and  problem  resolution, 

8.  Has  excellent  communication  skills  and  applies 
those  skills  to  each  audience  and  situation, 

9.  Knows  his  or  her  own  limitations  and  is  moti¬ 
vated  to  keep  a  safe  margin  above  that  limit,  and 

10.  Has  the  ego-strength  to  enforce  his  or  her  own 
limitations  in  every  situation. 

Responses  to  the  critical  incident  questions  pro¬ 
vided  numerous  examples  of  tasks  to  test  for  expertise 
in  this  domain  including:  reported  gear  and  gear 
indication  problems,  pressurization  abnormalities, 
aircraft  control  at  high  altitude  in  turbulence,  and 
system  operations  (auto-pilot,  avionics,  standby  gyros). 

From  these  characteristics  an  expert  aviator  analy¬ 
sis  was  performed  establishing  abstractions  about  the 
tasks  involved  in  expert  flying.  The  tasks  identified 
were  classified  as  skills,  knowledge,  and  mental  mod¬ 
els.  Most  of  these  components  were  found  to  be 
domain  specific  and  included  perceptual-motor  skills, 
procedural  skills,  and  knowledge  about  the  domain 
and  all  related  domains  (e.g.  weather,  air  traffic  con¬ 
trol,  physiology,  etc.).  Other  components  may  be 
domain  specific,  but  also  could  be  carried  across 
domains  in  expert  behavior.  These  qualities  are  mostly 
descriptive  of  the  mental  models  and  include  motiva¬ 
tion,  judgment  (decision-making skills),  and  commu¬ 
nication  skills.  Additional  traits  contributing  to  the 
ability  of  an  expert  aviator  to  exhibit  his  expertise  are 
those  of  maturity  in  thought  and  action,  consider¬ 
ation  of  man  and  machine,  honesty  to  self  and  others, 
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and  smoothness  in  control  of  the  aircraft.  These 
qualities  may  be  considered  more  style  than  factors  in 
expertise,  though  the  concepts  were  considered  im¬ 
portant  by  many  of  the  interviewees. 

Three  factors  were  found  to  be  important  in  distin¬ 
guishing  the  expert  pilot  from  the  "average”  pilots 
1)  their  method  of  information  acquisition,  2)  their 
decision  processes,  and  3)  their  communication  skills. 
From,  this  we  hypothesized  that  1)  the  expert  pilot  will 
seek  more  and  better  quality  information  about  the 
task  at  hand  and  will  seek  more  information  regarding 
the  changing  state  of  the  situation  than  the  competent 
pilot,  2)  the  expert  pilot  will  make  more  decisions 
than  competent  pilots,  and  3)  the  expert  pilot  will 
communicate  more  efficiently  with  all  of  the  potential 
resources  for  decision  making  than  the  competent 
pilot.  Study  3  was  then  designed  to  evaluate  these 
hypotheses. 


STUDY  3 

EXPERT  AVIATOR  ANALYSIS 

One  expert  mental  model  that  applies  to  all  do¬ 
mains  has  not  been  developed.  The  literature  indi¬ 
cates  that  expertise  is  highly  domain  specific  and  that 
it  is  unlikely  to  yield  a  model  that  can  be  applied  across 
domains.  The  literature  on  naturalistic  decision  mak¬ 
ing  indicates  that,  in  the  acquisition  of  expertise,  a 
transformation  of  qualitative  knowledge  organization 
takes  place  as  the  novice  progresses  to  expertise.  This 
suggests  that  expertise  in  any  domain  must  be  taught 
as  a  series  of  steps  leading  to  the  expert  mental  model. 
Thus  the  structure  of  the  expert  aviator  analysis  may 
consist  of  three  progressive  acquisition  phases:  knowl¬ 
edge,  skills,  and  mental  models.  The  initial  evidence 
in  our  investigation  of  the  expert  aviator  suggests  that 
the  input  and  output  for  the  “expert  model”  is  domain 
specific,  while  the  process  from  novice  to  expert  is 
general  across  most  aviation  domains. 

Following  Ryder  and  Redding’s  (1993)  process  on 
integrating  cognitive  task  analysis  into  instructional 
systems  development,  in  Study  3  a  novice  to  expert 
progression  analysis  was  performed  on  the  verbal 
responses  of  six  domain-experienced  pilots  using  a 


scenario  developed  specifically  to  test  for  the  charac¬ 
teristics  identified  in  the  previous  stage.  The  analyses 
of  each  of  the  seven  events  of  this  scenario  were 
established  through  close  adherence  to  aircraft  operat¬ 
ing  manuals,  standard  operating  procedures,  Federal 
Aviation  Administration  guidelines  and  rules,  pub¬ 
lished  training  manuals,  and  additional  materials  ap¬ 
plicable  to  the  task.  The  focus  of  the  analyses  was  on 
the  mental  models  which  were  used  to  establish  per¬ 
formance  criteria  for  each  subject  on  each  task.  These 
criteria  are  not  subjective  evaluations  of  the  outcome, 
but  measures  of  how  closely  the  subjects’  encoded 
responses  matched  our  model  of  the  expert  pilot. 

Simulation  Scenario  Development 

The  scenario  written  for  the  project  was  developed 
after  an  in-depth  study  of  the  accident  reports  from 
the  NTSB  on  the  three  domain  aircraft.  Accidents  in 
the  three  domain  aircraft  from  the  beginning  of  pro¬ 
duction  of  each  through  1992  were  reviewed.  Con¬ 
tributing  factors  to  the  accidents  were  compiled  into 
major  areas  to  be  used  in  the  simulation  scenario.  The 
most  significant  problems  identified  in  these  aircraft 
were:  weight  and  balance,  fuel  planning,  system 
abnormals  in  instrument  meteorological  conditions 
(IMG),  suspected  in-flight  break-up,  effects  of  icing, 
flight  into  adverse  weather  conditions,  and  controlled 
flight  into  terrain.  From  this  list  of  problems  we 
identified  those  that  could  be  simulated  easily  and 
also  would  discriminate  competent  pilots  from  ex¬ 
perts.  The  seven  events  that  we  identified  for  the 
simulation  scenario  were: 

1 .  Performing  a  weight  and  balance  assessment  and 
checking  fuel. 

2.  Landing  gear  abnormal  on  take-off. 

3.  Icing  conditions  during  climb-out. 

4.  Hypoxia  at  altitude. 

5.  Unexpected  holding  with  moderate  turbulence 
on  arrival. 

6.  Reported  windshear  conditions  on  approach. 

7.  Landing  gear  abnormal  on  landing. 

These  seven  events  were  then  assembled  into  our 
experimental  flight  scenario.  The  text  of  the  scenario 
is  presented  in  Appendix  C. 
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Event  Analysis 

Table  1  presents  an  analysis  of  the  factors  identify¬ 
ing  the  progression  from  novice  to  expert  for  the  seven 
scenario  events  used  in  the  Study  4  experiment.  These 
outlines  break-down  the  specific  experimental  events 
into  knowledge,  skills,  and  mental  models.  The  out¬ 
lines  progress  through  three  developmental  stages 
with  the  definitions  at  each  level  being  slightly  differ¬ 
ent.  The  knowledge  progression  begins  with  domain 
concepts,  rules,  and  procedures.  It  then  proceeds  to 
declarative  knowledge  structure,  and  then  to  expert 
knowledge  organization.  The  skills  advance  from  the 
basic  necessary  skills,  through  competent  skill  compo¬ 
nents,  to  the  refinement  of  these  skills.  Mental  models 
are  the  deductive  framework  for  problem  solving 
providing  the  structure  for  knowledge  and  skill  orga¬ 
nization  and  utilization  in  the  domain.  In  the  analysis 
of  data  in  Study  3,  ptesented  below,  we  subjectively 
determined  the  extent  to  which  each  subject’s  behav¬ 
ior  corresponded  to  the  level  of  expertise  identified  by 
these  factors  for  each  event. 

Table  2  presents  a  general  list  of  factors  identified 
with  aviator  expertise  in  this  exercise  based  on  situa¬ 
tion  assessment,  conditions,  and  prerequisite  infor¬ 
mation  specific  to  the  scenario.  Table  3  presents  a  new 
teformulated  model  contrasting  the  characteristics  of 
the  competent  and  the  expert  general  aviation  pilot. 
In  combination  these  factors  and  models  were  used  as 
criteria  for  evaluating  the  subject  pilots  in  Study  4. 

STUDY  4 

VERBAL  PROTOCOL  ANALYSIS 

The  fourth  study  of  the  project  consisted  of  a  verbal 
protocol  analysis  based  on  the  scenario  and  event 
analyses  developed  in  Study  3.  An  experimental  pro¬ 
tocol,  shown  in  Appendix  C  was  established  and 
administered  to  six  subject  pilots.  These  six  subjects 
had  not  participated  in  any  of  the  previous  studies. 
The  objective  of  this  experiment  was  to  gain  further 
insight  into  the  thought  processes  of  expert  pilots  in 
an  even  more  structured  simulation  scenario  and  to 
refine  our  model  of  the  expert  pilot  in  a  form  that 
could  be  tested  in  the  next  level  of  the  study. 


Subject  Selection 

The  subjects  for  the  verbal  protocol  analysis  were 
six  volunteer  pilots  from  Florida  and  Ohio.  Subjects 
were  selected  based  on  their  availability  and  recent 
experience  in  one  of  the  three  domain  aircraft.  Most 
of  the  subjects  were  qualified  in  two  or  more  of  the 
domain  aircraft.  Although  posting  and  word-of-mouth 
procedures  were  used  to  solicit  pilot  subject  volun¬ 
teers,  all  subject  pilots  were  fairly  well  know  to  the 
experimenter.  A  summary  of  the  demographic  infor¬ 
mation  about  the  subjects  is  shown  in  Table  4.  Subject 
Pilots  1  and  2  fly  for  Part  135  operators.  Subject  Pilot 
3  flies  for  a  Part  121  operator  and  Subject  Pilots  4,  5, 
and  6  fly  for  corporations. 

Procedures 

A  “flight  kit”  with  the  applicable  charts,  aircraft 
manuals,  supplies,  and  equipment  to  “fly”  the  sce¬ 
nario  was  brought  to  the  experimental  session  as 
shown  in  Appendix  C.  Each  subject  was  given  the 
opportunity  to  request  all  of  the  items  he  would 
“usually”  take  on  a  trip  of  this  type  in  the  domain 
aircraft  with  which  he  was  most  familiar. 

The  scenario  used  (described  earlier)  was  designed 
to  be  as  close  to  realistic  as  possible  through  the  use  of 
repeated,  actual  experiences  of  the  first  author  over 
the  same  route  of  flight  in  various  weather  conditions. 
The  weather  information  package  consisted  of  print¬ 
outs  of  actual  weather  obtained  for  the  experiment  on 
a  particular  day  for  the  flight.  The  flight  plans  were 
consistent  across  the  three  domain  aircraft,  changed 
only  to  reflect  the  type  of  aircraft  being  flown.  In  each 
case  the  subjects  “flew”  the  aircraft  in  which  they  were 
most  familiar. 

When  the  subjects  came  in  for  their  session,  the 
protocol  found  in  Appendix  C  was  administered, 
beginning  with  filling  out  the  human  subjects  ap¬ 
proval  form.  All  other  paperwork  was  completed  by 
the  experimenter.  The  experiment  was  conducted 
identically  in  all  cases  with  the  subject  completing  the 
experimental  scenario,  then  completing  the  Aviator 
Questionnaire.  Prompting  questions  were  used  when 
necessary  with  approximately  the  same  frequency  for 
all  subjects.  At  no  time  were  the  subjects  instructed  to 
“tell  me  why”  or  “tell  me  what  you  were  thinking.”  All 
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questions  were  either  for  clarification  of  a  statement 
or  to  continue  the  session.  The  entire  process  was  tape 
recorded  for  transcription.  All  of  the  subjects  were 
debriefed  after  the  session  and  were  given  the  oppor¬ 
tunity  to  express  comments  regarding  the  procedure. 
The  subjects  reported  that  the  process  was  both  real¬ 
istic  and  interesting. 

The  tape  recordings  of  the  sessions  were  tran¬ 
scribed  and  then  coded  into  categories  for  data  analy¬ 
sis.  Any  statement  that  could  not  be  understood  or 
was  not  clear  was  not  coded.  The  coding  was  com¬ 
pleted  by  the  experimenter  with  one  subject  also  being 
coded  by  an  independent  coder  to  check  for  coding 
reliability.  The  coding  categories  were  then  refined 
and  the  frequencies  were  compiled. 

Data  Coding 

The  initial  verbal  protocol  analysis  data  coding 
categories  were  determined  from  the  aviator  cognitive 
analysis  as  well  as  the  expert  aviator  models  developed 
in  Study  3.  After  completing  the  experimental  proto¬ 
col,  transcription  of  the  tapes,  and  encoding  of  the 
tapes,  the  coding  categories  were  further  revised  to 
better  reflect  the  information  being  presented  by  the 
subjects.  The  coding  categories  were  as  follows: 

1.  Total  Requests:  Total  number  of  inquiries  for 
supplies  and  all  types  of  information. 

1.1  Supply:  An  item  requested  by  the  pilot  to  be 
used  on  the  “flight”  such  as  aeronautical  charts, 
calculator,  flashlight,  etc. 

1 .2  Information:  This  category  is  a  total  of  all  of 
the  sub-categories  of  the  types  of  information  re¬ 
quested. 

1 .2.1  Weather:  Requests  about  the  weather  such  as 
“What  are  the  winds  again?”,  “...ask  them  to  check 
on  the  weather  and  see  how  bad  it  was....” 

1.2.2  Position:  Requests  regarding  the  subjects’  po¬ 
sition,  or  position  of  other  aircraft.  For  example, 
“. .  .see  what  kind  of  stack  up . .  .delays  for  other  planes 
in  the  que...”,  or  “..Try  to  get  my  new  current 
groundspeed  to  find  out  if  Fm  going....” 

1 .2.3  Clarification:  A  question  of  confirmatory  na¬ 
ture  such  as,  “It’s  called  Allegheny  County?”,  or 
“and  this  is  an  air  traffic  control  problem?” 

2.  Advise  ATC:  The  category  is  used  when  the 
subject  indicated  that  he  will  inform  air  traffic 


control  of  the  situation. 

3.  Situation  Evaluation:  When  the  subject  is  mak¬ 
ing  comments  (or  questions)  about  the  state  of 
the  aircraft,  weather,  air  traffic,  and  considering 
the  information  available.  “I  could  listen  to  the 
hydraulic  pump  and  if  the  pump  is  still  run¬ 
ning...”  and  “First  of  all,  I  want  to  find  out  how 
much  time  I  can  expect  to  hold,...” 

4.  Curiosity:  A  comment  indicating  the  subject  just 
wanted  to  know  why  in  detail. 

5.  Alternative  Generation:  The  verbalization  of  al¬ 
ternatives  such  as  “If  after  my  expect  further 
clearance  time,  I  would  hold  another  half  hour,  if 
any  longer,  I’ll  go  to  my  alternate.” 

6.  Action:  This  category  is  used  when  only  an  action 
is  stated  as  in  “put  the  gear  down.” 

7.  Total  Statements:  This  is  the  total  of  the  sub¬ 
categories  below. 

7. 1  Procedural:  Comment  on  a  procedure  such  as 
“well,  the  first  thing  you  do  is  get  out  supplemental 
oxygen  and  it’s  a  canister,  it’s  not  one  you  want  to 
use  unless  you  have  to...”,  or  “they  usually  figure  20- 
21  gallons  per  hour...” 

7.2  System:  A  statement  regarding  an  aircraft  or 
airspace  system.  “You  have  your  upper  and  lower 
door  to  consider”,  and  “the  mains  are  down  so  there 
is  no  little  mirror  to  check  the  nosegear...” 

7.2  Experience:  A  statement  that  based  on  the 
subjects’  experience,  past  or  during  the  experiment. 
“I  usually  file  direct  when  I  go  up  to  that  altitude”, 
“You  would  have  noticed  it  wasn’t  pressurized  as 
you  were  going  up,  I  would  have  noticed  that  right 
away”,  and  “the  forecast,  2,500  broken,  but  being  as 
it  is  down  now,  and  it  may  not  come  up..,,” 

7.4  Error:  A  statement  that  includes  erroneous 
information,  such  as  “I  don’t  have  the  minimums 
for  the  approach”. 

7.5  Readback:  The  repeating  of  a  direction  or 
statement  by  the  subject. 

7.6  Positional:  A  statement  on  one’s  position 
for  instance,  “so  I  would  be  coming  in  from  the 
northwest....” 

7.7  General:  “A  lot  of  thunderstorms”  or  “I 
would  have  highlighted  this  ahead  of  time...”  are 
statements  that  are  not  specific  enough  to  fall  in 
any  other  category. 
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Table  1.  Cognitive  task  analysis  outline  of  seven  flight  events  as  a  function  of  level  of  pilot  competency. 
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Maneuvering  Recognition  of  Limitations  on  Holding  patterns  Procedural  Use  of 
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Air  traffic  control 
system 


Table  2.  Factors  making  up  the  general  expert  mental  model  of  aviators 


SITUATION  ASSESSMENT 


Aircraft  Condition  Data 
System  Condition 
Fuel  State 
Abnormals 
Emergencies 
Airframe  Status 
Engine  Status 


Position  Data 
Altitude 
Airspeed 
Configuration 


Flight  Plan 
As  Filed 
Deviations 
On  Schedule 


CONDITIONS 


Weather  Factors 
Winds 
Turbulence 
Icing 

Thunderstorms 

Temperature 

Dewpoint 


Navigation  Facilities 

VOR 

ADF 

GPS 

DME 

LORAN 

ILS 


Pilot  Factors 
Fatigue 

Personal  Factors 
Crew  Coordination 
Personal  Limitations 


Air  Traffic  Control 
Volume  of  Traffic 
Potential  Conflicts 


Aircraft  Factors 
Current  Condition 
Aircraft  Limitations 


Skills 

Gross  motor  skills 
Perceptual-motor 
Perceptual 
Procedural 
Decision  Making 
Time  sharing 
Skill  Integration 
Limitations 


PREREQUISITE  INFORMATION 

Knowledge  Decision-Making  Skills 

Aircraft  Problem  Solving  Strategies 

Environment  Creativity 

Pilot  (self)  Prioritization 

Limits  Compartmentalization 

Time  Management  Curiosity 
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Table  3.  Model  of  the  competent  and  expert  general  aviation  pilots 


COMPETENT  PILOT  EXPERT  PILOT 

KNOWLEDGE 

Knows  the  domain  sufficiently  to  pass  Knows  the  domain 

FAA  exams  Knows  him/herself 

Knows  the  environment 
Knows  the  organization 


SKILLS  OR  ABILITIES 

Skills  and  abilities  sufficient  to  pass  FAA  exams  Highest  technical  skill 

Superior  mental  abilities  for  problem 
diagnosis,  risk  assessment,  and 
problem  resolution 
Ability  to  focus  attention 
Ability  to  change  focus  of  attention 
Adaptable  communication  skills 


BEHAVIOR 

Usually  follows  FARs  Avoids  situations  that  push  skill 

Takes  the  BFR  and  IFR  proficiency  tests  Keen  observer  of  the  flight  environment 

Establishes  baseline  for  normal  operations 
Makes  contingency  plans 
Works  continuously  to  improve  knowledge, 
skill,  and  abilities 


MOTIVATION 

To  continuously  learn  about  domain 
To  be  skeptical  about  “normal”  situation 
To  overcome  pressures  to  push  risk 
To  change  focus  of  attention  when  needed 


May  be  primarily  focused  on  matters  outisde 
of  cockpit 


Table  4.  Pilot  demographics 


SI 

S2 

S3 

S4 

S5 

S6 

Age 

26 

24 

50 

55 

54 

49 

Flying  Years 

8 

5 

29 

25 

5 

10 

Total  Time 

2600 

2600 

10600 

14000 

2000 

2200 

Instrument  Time 

130 

110 

850 

300 

200 

600 

Domain  Aircraft  Time 

600  (C) 

500  (C) 

35  (P) 

350  (B) 

200  (C) 

115 (P) 

(C=Cessna  210,  P=Piper  Malibu,  B=  Beech  Baron) 


21 


8.  Decisional:  A  comment  which  states  or  implies  a 
decision,  or  decision  process  such  as  “Since  I  had 
trouble  with  the  shimmy  dampener,  I  would  prob- 
ably  figure  it  wasn't  a  false  indication...”,  “I  don't 
know  if  I’d  go  in  this  weather,  tornadoes,  thunder¬ 
storms...”,  and  “There’s  some  amendments  in  there, 
so  may  have  to  worry  about  it  changing  as  we  go 
down  there,  but  generally  it's  workable.” 

9.  Decision  Process:  This  category  usually  consisted 
of  a  lengthy  evaluation  and  explanation  of  a  situa¬ 
tion  outlining  the  plan  or  system  used  in  making  a 
decision.  This  differs  from  the  Statement,  Deci¬ 
sional  in  the  complexity  of  the  discussion  and  num¬ 
ber  of  alternatives,  statements,  and  evaluations  made 
in  succession.  An  example,  “Well,  first  consider¬ 
ation  would  be  is  if  we  encounter  windshear  on  the 
approach,  execute  the  missed  approach  and  proceed 
to  the  alternate,  have  time  check  out  the  alternate 
weather  and  come  back  to  Charlotte  when  it  is 
suitable,  That  would  be  my  plan.  I'd  go  ahead  and 
attempt  the  approach,  but  at  first  indication  of 
windshear  that  was  unacceptable  for  this  aircraft,  I'd 
go  ahead  and  skip  this  and  go  to  the  alternate.” 

10.  Decision:  This  is  the  number  of  instances  in  which 
it  was  clear  that  a  decision  has  been  made  as  in  “I  will 
come  back  and  land,”  or  “Then  it  was  down  and  I'd 
go  back  to  Charlotte.” 

11.  Total  Decision  Making:  This  is  the  sum  of  the 
Statement-  Decisional,  Decision  Process,  and  Deci¬ 
sion  categories  for  use  in  data  analysis.  The  purpose 
of  the  category  is  to  look  at  the  total  number  of 
events  concerned  with  decision  making. 

12.  Expert  Rating:  This  is  a  subjective  rating  made  by 
the  experimenter  of  how  closely  the  subjects'  known 
flying  behavior  approximates  that  of  the  10  charac¬ 
teristics  of  the  expert  aviator  defined  in  Study  2. 
Although  it  would  have  been  desirable  to  have 
multiple  raters,  only  one  was  used.  This  decision  to 
use  only  one  rater  reflects  both  the  logistical  diffi¬ 
culties  of  locating  other  raters  in  the  various  places 
where  data  were  gathered  and  the  exploratory  na¬ 
ture  of  this  study.  Given  the  nature  of  the  study  and 
the  overall  intent  of  identifying  as  opposed  to 
measuring  components  of  the  pilot  expertise  model, 
the  use  of  single  rater  provides  a  sufficient  though 
certainly  not  optimal  approach. 


Data  Analysis 

Phrases,  ideas  and  statements  were  encoded  from 
the  transcripts  and  compiled  for  the  coding  categories 
listed  above.  The  primary  goal  of  the  analysis  was  to 
show  the  level  of  correspondence  between  the  hypoth¬ 
esized  model  of  the  expert  aviator  and  the  data  from 
these  pilot  experts.  Specifically,  the  three  hypotheses 
that  were  tested  were:  1)  Expert  pilots  seek  more 
quality  information  in  a  more  timely  manner,  (i.e. 
they  know  what  to  ask  for  and  when  to  ask  for  it)  than 
competent  pilots,  2)  Expert  pilots  make  more  deci¬ 
sions  than  competent  pilots,  3)  Expert  pilots  commu¬ 
nicate  more  readily  with  all  available  resources. 

These  three  areas  were  expected  to  contribute  in¬ 
sight  into  decision  making  and  thought  processes  as 
observed  during  the  problem  solving  experiment.  The 
first  analysis  examined  the  number  of  statements 
made  in  each  category  by  each  pilot  subject.  The 
second  analysis  was  a  subjective  evaluation  by  the 
experimenter  of  each  pilot  subject  based  on  indepen¬ 
dent  knowledge  of  the  pilot. 

Results 

To  evaluate  the  first  hypothesis  (information  gath¬ 
ering)  the  type  and  number  of  requests  for  informa¬ 
tion  or  material  made  by  the  subjects  was  used.  As 
shown  in  Figure  2,  Subject  3,  and  to  a  lesser  degree 
Subject  5,  have  a  noticeably  larger  number  of  infor¬ 
mational  requests  than  the  other  subjects.  On  closer 
observation  of  each  individual  subject  with  respect  to 
task,  these  same  subjects  also  seemed  to  ask  more 
appropriate  questions  in  each  of  the  circumstances. 

The  second  hypothesis  (decisions)  was  evaluated 
initially  using  the  Decision  frequencies.  In  this  case, 
the  data  show  that  Subjects  3  and  6  made  the  most 
decisions.  To  provide  a  more  robust  score,  it  is  useful 
to  combine  the  three  categories  of  Decisional  State¬ 
ment,  Decision  Process,  and  Decision  into  a  Total 
Decision  score.  Using  this  Total  Decision  score, 
Subjects  3  and  6  have  considerably  higher  scores  than 
the  others  (See  Figure  3).  These  two  subjects  tended 
to  verbalize  their  decision-making  processes  in  much 
greater  detail  than  the  other  four  subjects.  These 
subjects  also  used  a  type  of  dynamic  problem  solving 
which  helped  to  direct  us  toward  our  final  model  of 
the  expert  pilot  (see  Figure  6),  including  satisficing, 
feedback,  and  keeping  their  options  open. 
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To  examine  the  third  hypothesis  (communica¬ 
tions),  type  and  number  of  statements  made  through¬ 
out  the  simulated  flight  were  used.  Again,  Subjects  3 
and  6  made  more  Total  Statements  (see  Figure  4).  All 
types  of  communication  (verbal,  non-verbal,  tactile) 
are  critical  in  pilot  duties.  The  pilot  judgment  litera¬ 
ture  and  cockpit  resource  management  training  courses 
emphasize  communication  skills  as  one  of  the  primary 
components  of  a  safe  and  effective  pilot  (Jensen, 
1995).  Although  the  data  from  this  study  suggest  that 
being  able  to  verbalize  the  rationale  for  actions  and  a 
propensity  for  seeking  (verbally)  more  information 
are  associated  with  pilot  expertise  (as  defined  here), 
generalizing  from  such  a  small  sample  is  clearly  a  risky 
affair.  Further,  other  studies  of  expertise  using  other 
domains  (Ericsson  and  Smith,  1991)  have  generally 
found  that  experts  are  unable  to  verbalize  the  reasons 
for  their  actions  because  of  the  high  degree  of  automa- 
ticity  that  has  developed.  The  data  collected  here  do 
not  provide  us  with  a  means  to  explain  why  the 
character  of  expertise  in  aviation  should  differ  from 
the  character  of  expertise  in  other  domains.  The 
present  results  are  suggestive,  but  certainly  not  com¬ 
pelling,  and  will  be  explored  further  in  subsequent, 
more  closely  controlled  studies  of  larger  groups  of 
expert  and  novice  pilots. 

“Expert”  Rating 

The  second  method  for  determining  expertise, 
sometimes  used  in  the  literature  is  the  peer  rating 
(Ericsson  and  Smith,  1991).  In  this  study  an  expert 
observer  (the  experimenter),  who  was  also  a  peer  of  all 
six  subjects,  provided  a  subjective  “expertise”  rating 
on  each  of  the  10  characteristics  of  the  expert  aviator 
developed  in  Study  2:  Self-Confidence,  Motivation, 
Practice,  Focused,  Situation  Awareness,  Information 
Seeking,  Prompt  Action,  Vigilant,  Communication, 
Sets  Limits.  Each  of  the  subjects  was  rated  on  a  scale 
of  1  (lowest)  to  10  (highest)  for  each  of  these  ten 
factors.  The  experimenter  used  information  known 
about  each  subject  independently  from  the  experi¬ 


mental  data  to  make  these  ratings  and  performed  the 
ratings  prior  to  the  experimental  data  collection. 
Table  5  shows  how  each  of  the  six  subjects  was  rated 
on  each  of  the  ten  factors.  A  summary  of  the  condi¬ 
tions  considered  in  the  expert  model  vs.  the  expertness 
rating  is  depicted  in  Figure  5. 

Although  all  six  subjects  were  rated  high  in  most 
categories,  Subjects  3  and  6  were  rated  more  highly 
than  the  other  four  on  this  single-rater  peer  rating 
scale.  These  data  tend  to  confirm  the  more  objective 
verbal  protocol  results  shown  in  Figures  3  and  4.  As 
one  would  expect,  there  appears  to  be  no  relationship 
between  the  expert  rating  and  either  flying  experience 
or  number  of  flying  years  (See  Table  4). 

In  this  final  study  an  attempt  was  made  to  differen¬ 
tiate  among  pilots  who  were  known  to  be  above 
average  using  both  verbal  protocol  analysis  of  a  verbal 
flight  simulation  and  expert  ratings.  The  results  sug¬ 
gest  that  one  can  discriminate  among  the  pilots  using 
this  procedure.  However,  an  open-ended  verbal  simu¬ 
lation  leaves  questions  concerning  the  use  of  verbal 
communications  as  a  measure  of  expertise.  Although 
pilots  are  taught  to  communicate,  and  expert  pilots  do 
communicate  very  well,  this  is  not  the  only  criterion 
for  pilot  expertise  as  shown  in  the  final  model. 

The  results  tend  to  indicate  that  pilots  have  differ¬ 
ent  methods  and  styles  for  solving  fairly  common 
flying  situations,  and  these  methods  are  not  related  to 
the  total  flight  time  or  number  of  years  flying.  Sub¬ 
jects  3  and  6,  who  were  rated  the  highest,  consistently 
made  initial,  rapid  assessments  of  situations  and  then 
proceeded  to  confirm  or  disprove  their  theories.  The 
other  subjects  usually  asked  for  some  other  type  of 
information,  or  made  other  unrelated  statements  be¬ 
fore  handling  the  presented  problem.  Finally,  these 
two  subjects  were  far  more  verbal  and  tended  to  talk 
in  paragraphs  while  the  others  typically  offered  only 
one  or  two  short  phrases.  Although  this  may  reflect 
different  personality  styles,  the  content  of  communi¬ 
cation  from  these  two  subjects  was  more  consistent 
with  the  expert  model  than  that  of  the  other  subjects. 
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Table  5.  Subjective  rating  of  expertise  for  the  six  subjects 


Factor 


Subjects 
S3  S4 


Self-Confidence 

Motivation 

Practice 

Focused 

Situation  A\A/areness 
Information  Seeking 
Prompt  Action 
Vigilance 
Communication 
Sets  Limits 


SUMMARY  AND  CONCLUSIONS 

The  objective  of  these  four  studies  was  to  develop 
a  model  of  the  expert  pilot  decision  maker  and  to  show 
how  it  could  be  used  to  develop  intervention  strategies 
to  improve  safety  in  the  target  population  of  mid¬ 
altitude  general  aviation  pilots.  To  accomplish  this 
objective,  NTSB  accident  reports  were  analyzed  for 
the  domain  aircraft,  the  relevant  literature  on  decision 
making,  expertise,  and  modeling  was  reviewed,  and 
four  studies  were  conducted  including  semi-struc- 
tured  interviews,  structured  interviews,  cognitive  task 
analysis,  and  verbal  protocol  analysis  of  a  simulation 
experiment.  Following  each  of  these  studies,  new 
insight  was  gained  and  modifications  were  made  to 
the  model  of  the  expert  pilot  decision  maker. 

The  results  of  this  series  of  studies  suggest  that 
expertise  in  general  aviation  may  have  very  little 
relationship  with  flight  time  after  a  certain  number  of 
hours  (perhaps  as  low  as  2,000  hours).  In  general  the 
primary  characteristic  distinguishing  the  expert  from 
the  competent  is  judgment.  More  specifically,  we 
propose  that  expertise  in  general  aviation  pilots  can  be 
defined  in  terms  of  the  following  ten  characteristics: 

1.  Self-confidence  in  his  or  her  skills  as  a  pilot, 

2.  Motivation  to  learn  all  there  is  to  know  about  the 
flight  domain  and  practices  their  skills  constantly, 

3.  Ability  to  focus  on  the  necessary  task  and  change 
that  focus  at  the  slightest  hint  that  a  change  is 
needed, 

4.  Situation  awareness  (flight  environment,  loca¬ 
tion  of  other  aircraft,  terrain,  navigation,  com¬ 
munications,  weather,  etc.), 

5.  Cognizant  of  the  machine  including  noise,  vibra¬ 
tion,  and  engine  indications, 

6.  Vigilant  for  the  unusual,  abnormal,  or  emer¬ 
gency,  and  mentally  makes  contingency  plans, 

7.  Mental  capacity  for  problem  diagnosis,  risk  as¬ 
sessment,  and  problem  resolution, 

8.  Communication  skills  and  applies  those  skills  to 
each  audience  and  situation, 

9.  Knowledge  of  his  or  her  own  limitations  and  moti¬ 
vation  to  keep  a  safe  margin  above  those  limits,  and 

10.  Ego-strength  to  enforce  his  or  her  own  limita¬ 
tions  in  every  situation. 


Based  upon  the  data  from  the  studies  described 
here,  and  taking  into  account  the  large  body  of  re¬ 
search  on  pilot  performance  along  with  his  own  in¬ 
sights  into  pilot  decision  making,  Jensen  (1995) 
proposed  a  general  model  of  the  expert  pilot  which  is 
shown  in  Figure  6.  The  characteristics  of  expertise  in 
general  aviation  pilots  suggested  above  might  be  de¬ 
scribed  in  terms  of  experience,  risk  management, 
problem  solving  and  attentional  control  and  aggre¬ 
gated  into  Jensen’s  larger,  more  general  model  of  the 
expert  pilot.  In  that  model  the  major  factors  con¬ 
tributing  to  pilot  expertise  which  subsume  the  find¬ 
ings  of  these  studies  are: 

Aviation  Experiences,  Proponents  of  the  expertise 
approach  to  studying  human  behavior  have  indicated 
that  expertise  is  gained  through  years  of  experience. 
They  often  say  that  10  years  of  experience  dedicated 
to  one  field  makes  an  expert.  Some  have  used  hours  of 
experience  in  flight  as  an  indication  of  expertise. 
Jensen  believes,  as  do  almost  all  other  researchers  in 
this  field,  that  it  takes  more  than  hours  of  flying 
experience  to  make  an  expert  pilot.  In  this  step  he 
proposes  that  there  are  five  aspects  of  experience  that 
are  necessary  to  fulfill  this  part  including,  number  of 
hours,  variety,  meaningfulness,  relevance,  and  recency. 
Many  hours  of  flight  in  Ohio  does  not  make  an  expert 
pilot  in  mountainous  terrain  —  a  variety  of  experi¬ 
ences  is  need.  Experiences  that  have  no  meaning  (e.g,, 
boring  lectures)  do  not  change  behavior  —  the  expe¬ 
riences  need  to  be  presented  in  ways  that  are  meaning¬ 
ful  to  the  learner  (e.g.,  experiences  with  sound,  visual 
effects,  motion  —  simulation).  The  experiences  must 
be  relevant  to  the  kind  of  flying  that  is  anticipated  of 
the  expert  (e.g,,  in  teams  if  for  a  multi-person  crew 
aircraft).  Finally,  the  experiences  must  be  recent. 
Although  some  experiences  remain  in  one’s  mind  for 
life,  most  need  to  be  reinforced  periodically  to  be 
available  for  expert  decision  making. 

Risk  Management.  The  second  step  in  creating  the 
expert  pilot  decision  maker  is  to  develop  risk  manage¬ 
ment  techniques.  This  step  requires  the  establishment  of 
a  proper  set  of  values  consistent  with  societal  norms  (i.e., 
the  pilot  must  know  the  safety  expectations  of  his  passen¬ 
gers  and  company).  It  also  requires  that  the  pilot  studies 
carefully  all  of  the  possible  hazards  and  the  probability 
that  these  hazards  could  affect  his  flight  practice. 
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AVIATION  EXPERIENCES 


Figure  6.  Jensen  (1995)  model  of  the  expert  pilot  decision  maker. 


Dynamic  Problem  Solving.  The  third  step  is  to 
develop  a  technique  for  solving  dynamic  problems 
known  as  satisficing.  This  technique  for  solving  ill- 
defined  problems,  which  has  been  identified  in  ex¬ 
perts,  consists  of  beginning  with  a  clear  understanding 
of  the  situation  and  making  decisions  that  have  a  good 
chance  of  leading  to  the  optimum  solution,  always 
keeping  the  safest  options  open.  Feedback  from  the 
action  is  used  to  gain  further  information  leading  to 
additional  decisions.  Finally,  the  optimum  choice  is 
made.  The  key  is  to  make  decisions  that  avoid  closing 
options. 

Attentional  Control .  The  final  step  is  to  develop  an 
ability  to  control  attention  so  as  to  focus  on  the  task 
at  hand  leaving  all  other  matters  out  of  mind.  In 
conjunction,  the  expert  pilot  must  also  be  able  to 


perceive  the  smallest  indication  that  something  else  in 
the  cockpit  deserves  his  attention  and  switch  to  that 
matter  quickly  and  deliberately.  It  means  being  able  to 
put  matters  outside  of  the  cockpit  out  of  your  mind  as 
well.  Finally,  it  means  not  allowing  pressures  to  make 
decisions  based  on  non-aviation  concerns  to  influence 
you  away  from  the  safe  choices  for  which  you  are 
committed  (Jensen,  1995). 

This  model  provides  a  framework  for  understand¬ 
ing  the  results  of  the  current  series  of  studies.  It  also 
provides  a  rational  platform  for  planning  further 
studies  directed  at  a  better  understanding  of  pilot 
cognition  and  behavior.  From  that  understanding  we 
believe  will  come  new  training  interventions  to  pro¬ 
duce  safer  general  aviation  pilots. 
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APPENDIX  A 


INITIAL  SEMI-STRUCTURED  INTERVIEW  QUESTIONNAIRE 

The  Ohio  State  University  Department  of  Aviation  is  investigating  enhanced  training  methods  for  the 
Federal  Aviation  Administration  through  FAA  Contract  No.  DTFAOl-92'10204.  We  are  asking  your 
cooperation  in  this  brief  interview  to  assist  in  the  definitions  and  terms  to  be  used  in  this  project.  The  personal 
data  and  information  you  provide  is  confidential  and  will  only  be  used  to  contact  you  in  the  future  if  you 
would  like  to  participate  in  another  phase  of  this  project.  Your  time  and  comments  are  most  appreciated. 

1.  How  can  you  distinguish  how  “good”  a  pilot  is? 

2.  If  you  were  to  group  pilots  into  three  categories  based  on  how  “good”  they  are,  what  would  you  name  the 
categories? 


a. 

b. 

c. 


3.  What  did  you  base  your  categories  on? 

4.  Describe  a  type  “a”  pilot.  Can  you  name  three? 

5.  Describe  a  type  “b”  pilot.  Can  you  name  three? 

6.  Describe  a  type  “c”  pilot.  Can  you  name  three? 

7.  What  is  the  best  maneuver  to  use  to  determine  how  “good”  a  pilot  is?  Why  did  you  choose  this  one? 

8.  Name  the  three  most  memorable  events  you  have  had  as  a  crewmember  in  any  aircraft. 

9.  Would  you  be  willing  to  participate  in  future  phases  of  this  project? 


A1 


APPENDIX  B 


AVIATOR  STRUCTURED  INTERVIEW 
AVIATOR  QUESTIONNAIRE 

Dr.  Richard  S.  Jensen,  myself,  and  The  Ohio  State  University  Department  of  Aviation  is  investigating 
enhanced  training  methods  for  the  Federal  Aviation  Administration  through  FAA  Contract  No.  DTFAOl'92- 
10204.  We  are  asking  your  cooperation  in  this  brief  interview  to  assist  in  the  definitions  and  terms  to  be  used 
in  this  project.  The  personal  data  and  information  you  provide  is  confidential  and  no  information  associated 
with  you  or  your  responses  will  be  released.  This  information  will  only  be  used  for  our  research  purposes,  and 
to  contact  you  in  the  future  if  you  would  like  to  participate  in  another  phase  of  this  project.  Your  time  and 
comments  are  most  appreciated. 

The  questionnaire  is  given  verbally  and  your  responses  will  be  recorded  and  transcribed  at  a  later  date.  There 
is  a  standard  consent  form  we  as  you  to  read  and  sign  before  you  participate  in  the  project.  If  at  any  time  you 
wish  to  discontinue  participation  in  this  study  for  any  reason,  simply  state  you  would  like  to  stop,  and  you  may 
do  so  without  question.  I  will  be  glad  to  answer  any  questions  you  have  before  or  after  the  interview. 

1.  Why  did  you  become  a  pilot? 


2.  What  are  your  best  attributes  as  a  pilot? 

3.  What  kinds  of  experiences,  either  in  training  or  in  other  instances  have  made  you  a  better  pilot? 

4.  Name  the  three  most  important  skills  that  are  components  of  your  task  as  a  pilot. 

5.  Describe  two  events  you  consider  to  have  been  critical  in  nature  while  flying. 

6.  How  might  you  have  responded  to  the  above  situations  differently? 

7.  What  one  factor  makes  a  pilot  above-average  compared  to  other  pilots  of  their  status? 

8.  What  is  the  most  difficult  aspect  of  flying  a _ (P-210,  Malibu,  P-Baron)? 
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SUBJECT  INFORMATION 


DATE  OF  INTERVIEW:  INTERVIEWER: 

AGE:  TOTAL  YEARS  FLYING  (Full  Time) :  CURRENT:  YES  NO 

WOULD  YOU  BE  WILLING  TO  PARTICIPATE  IN  OTHER  PHASES? 

WOULD  YOU  BE  WILLING  TO  FLY  IN  A  SIMULATOR? 

NAME _ 

ADDRESS _ 

TELEPHONE  (H) _ (W) _ 

EMPLOYER _ 

PILOT  CERTIFICATES/RATINGS _ 

AIRCRAFT  OWNED  (CURRENT) _ (PREVIOUS) 

FLIGHT  TIMES: 

TOTAL _ INSTRUMENT _ NIGHT _ 

TIME  IN  TYPE: 

CESSNA  P-210 _ PIPER  MALIBU _ BEECH  BARON  _ 

TRAINING:  LAST  TRAINING  SESSION  (date  and  place) _ 

TIME  SPENT  IN  AVIATION  ACTIVITIES  PER  MONTH _ 

PURPOSE  OF  FLYING: 

_ %FOR  HIRE  (type): _ %PERSONAL _ %BUSINESS _ %FUN 
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APPENDIX  C 


EXPERIMENTAL  PROTOCOL 


Experimental  Kit  Checklist: 

1  Jeppesen  Sandersen  Flight  Case 

Jeppesen  and  NOAA  Low  and  High  Altitude  Enroute  Charts 
Jeppesen  and  NOAA  SIDS,  STARS,  and  Approach  Charts 
Applicable  Sectional  Charts  (Detroit,  Cincinnati,  Charlotte) 

Airman’s  Information  Manual 
Federal  Aviation  Regulations 

1  D-Cell  Flashlight 

2  E6B  Flight  Computers 

Aircraft  Flight  Manuals  (Cessna  1978  Pressurized  Centurion,  Piper  Malibu,  Beechcraft  Pressurized  Baron  58P 
and  58PA) 

1  Calculator 

1  Voice  Activated  Tape  Recorder  (with  spare  tapes) 

Pens  and  Pencils 

2  Copies  of  Written  Weather 

Flight  Plans  and  Flight  Logs  for  C-210P,  PA-46P,  and  B-58P  ) 

Human  Subject  Consent  Forms 
Subject  Information  Forms 
Verbal  Protocol  Scenario  Script 


VERBAL  PROTOCOL 


SUBJECT  PRE-BRIEFING 

“Dr.  Richard  S.  Jensen,  myself,  and  additional  members  of  The  Ohio  State  University  Department  of  Aviation 
are  investigating  enhanced  training  methods  for  the  Federal  Aviation  Administration  through  Contract  No. 
DTFAOl -92-1 0204.  Thank  you  for  agreeing  to  participate  in  our  study. 

This  portion  of  our  study  will  involve  reading  you  a  common  situation  you  may  encounter  while  flying  your 

- .  This  session  should  not  last  any  longer  than  2  hours.  The  background  of  this  session  is  a  flight 

from  AGC  (Pennsylvania)  to  CLT  (North  Carolina)  for  a  business  meeting.  Before  we  begin,  you  will  have  the 
opportunity  to  request  information,  charts,  equipment,  and  supplies  that  you  would  normally  take  on  an  IFR 

flight  in  your _ .  There  are  six  different  flight  segments  that  I  will  read  to  you.  After  listening  to 

the  segment,  I  would  like  you  to  tell  me  in  detail  what  you  would  do. 

The  session  will  be  tape  recorded,  but  as  explained  on  the  consent  form,  your  name  will  not  be  used  or  kept  with 
the  information  from  the  study.  The  responses  and  data  that  you  provide  is  confidential,  and  no  information 
associated  with  you  or  your  responses  will  be  released.  If  at  any  time  you  have  a  question  regarding  the  study, 
please  feel  free  to  ask.  If  at  any  time  you  wish  to  discontinue  participation  in  this  study  for  any  reason,  simply 
state  you  would  like  to  stop,  and  you  may  do  so  without  question.  After  the  session,  we  will  debrief  in  full  and 
review  the  flight  segments.  Do  you  have  any  questions  before  we  begin?” 
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FLIGHT  REGIME  FOR  SCENARIO  DEVELOPMENT 


The  following  flight  segments  were  selected  from  a  theoretical  flight  from  the  Allegheny  County  Airport 
(Pittsburgh,  Pennsylvania)  to  the  Charlotte-Douglas  International  Airport  (CLT)  in  Charlotte,  North  Carolina. 
This  flight  was  to  take  place  in  the  summer  with  marginal  weather  at  AGC  and  deteriorating  weather  enroute  to 
CLT  due  to  a  fast  moving  cold-front  approaching  the  Appalachian  mountains.  The  route  of  flight  and  weather 
pattern  development  were  constructed  to  facilitate  realism  in  the  flight  segments  that  follow. 

The  subjects  in  the  experiment  will  be  given  a  prepared  flight  plan  (IFR)  and  an  opportunity  to  make  changes  to 
the  plan.  They  will  be  allowed  to  request  the  necessary  equipment,  charts,  and  other  information  they  would 
normally  have  on  a  trip  of  this  type.  The  background  information  given  to  the  subjects  includes  a  brief  discussion 
of  the  flight.  Supplies  requested  that  are  not  available  will  be  recorded  as  such.  Decision  trees  and  analysis  will  be 
based  on  consistent  procedures  and  information  from  the  experlmenter/”air  traffic  controller”.  The  available 
equipment  and  information  is  listed  below  followed  by  the  flight  plan,  flight  log,  and  weather  for  the  scenario. 

Supplies  Available  to  Subjects: 

1.  Pilot  Operating  Handbook  for  Cessna  P-210,  Piper  PA-46,  or  Beechcraft  P-58. 

2.  Low  and  High  altitude  enroute  charts,  either  NOS  or  Jeppesen. 

3.  Sectional  and/or  WAC  charts. 

4.  Approach  charts,  either  NOS  or  Jeppesen. 

5.  Flashlight, 

6.  Flight  plan  (already  completed). 

7.  Flight  log  (already  completed). 

8.  Weather  -  printed  SA,  FT,  FA,  winds  aloft,  NOTAMS,  and  SIGMETS. 

9.  Information  regarding  aircraft  equipment  and  status. 

10.  Airman’s  Information  Manual/Federal  Aviation  Regulations. 


Flight  Plan: 

IFR  flight,  N527JK,  PA-46,  C-210P,  B-58P,  all  /R  (RNAV  equipped),  200kts  true  airspeed,  departing  AGC,  at 
1600  UTC,  cruising  FL200,  route  of  flight  filed  direct,  CLT.  Time  enroute  2  hours,  fuel  on  board  4  hours, 
alternate  GSO,  4  souls  on  board,  aircraft  is  white  with  red. 

Flight  Log: 

Although  the  flight  is  filed  “direct”,  the  route  of  flight  proceeds  from  Allegheny  County  (AGC),  Parkersburg 
West  Virginia  (JPU),  Charleston,  West  Virginia  (HVQ),  Holston  Mountain  (HMV),  the  Shine  5  arrival  into 
Charlotte  (CLT). 
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Weather  for  Flight  Scenarios 

Actual  printed  weather  packets  were  issued  to  each  participant  in  the  study  using  standard  NOAA  format. 

PRE-FLIGHT  TASK  -  Weight  and  Balance  and  Fuel  Calculations 

Prior  to  starting  the  scenario,  the  subjects  are  told  that  the  flight  plan  and  flight  log  have  already  been  prepared 
(including  four  hours  of  fuel).  They  are  asked  to  “check  your  weight  and  balance”  (there  are  four  souls  on  board 
at  170  pounds  each  and  25  pounds  of  baggage  each)  and  asked  if  they  want  to  make  any  changes  to  their  flight 
plan  or  fuel  loading.  The  tape  recorder  is  voice-activated,  so  a  note  of  the  preparation  time  is  made  for  each 
subject. 

After  the  subject  completes  the  pre-flight  tasks  as  requested,  the  scenario  segments  are  started.  The  text  of  the 
segments  follow.  Statements  in  parenthesis  are  notes  for  the  experimenters’  use  only. 

TASK  1  -  Nosewheel  Shimmy  on  Take-Off 

“You  are  departing  on  RWY  23  at  AGC.  Just  before  rotation  there  is  a  considerable  nosewheel  shimmy.  After 
retracting  the  landing  gear,  the  red  ‘gear  unsafe’  light(s)  remain  illuminated. 

How  will  you  handle  this  situation?” 

TASK  2  -  Airframe  Icing 

“You  are  climbing  through  FL  180  and  have  been  IMG  since  passing  through  4000  feet  MSL.  You  notice  the 
aircraft  is  not  climbing  as  well  as  it  should  for  the  weight.  You  also  notice  a  strange  whistling  sound  and  are  not 
able  to  determine  where  it  is  coming  from. 

What  are  your  actions?” 


TASK  3  -  Subtle  Cabin  Pressure  Loss  at  Altitude 

“You  have  been  cruising  at  FL  200  for  about  an  hour.  The  weather  is  still  IMG,  but  it  is  now  -9  degrees  C  and 
the  ice  accumulation  has  stopped.  The  unforecast  headwinds  and  vectors  you  have  been  given  has  added  about 
20  minutes  to  your  flight  time.  You  feel  your  ears  popping  and  wish  you  could  get  rid  of  a  nagging  headache. 
You  are  given  a  frequency  change  to  Indianapolis  Center  and  fumble  with  the  radio  controls.  After  contacting 
Center,  you  find  yourself  laughing  at  the  controllers’  instructions  to  turn  30  degrees  left  for  traffic.  You  are  on 
autopilot,  initiate  the  turn  and  finally  scan  your  gauges.  It  seems  to  take  a  long  time  to  comprehend  what  they 
are  reading. 

What  will  you  do  now?” 

(To  what  altitude  will  you  descend?) 
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TASK  4  -  Unexpected  Hold  (including  moderate  to  severe  turbulence) 


“You  are  approaching  the  HMV  VOR  and  thinking  to  yourself  that  it  hardly  pays  to  file  direct  anymore.  You 
have  been  vectored  all  over  the  place,  the  weather  is  not  the  greatest,  and  apparently  you  are  catching  up  to  that 
cold-front  as  the  ride  is  getting  pretty  rough.  You  have  to  ask  your  front  seat  passenger  to  hold  your  charts  as  they 
will  not  stay  on  the  chart  holder.  Center  calls  with  an  amendment  to  your  routing  when  you  are  ready  to  copy  as 

follows: _ ,  you  are  now  cleared  direct  to  the  HMV  VOR,  hold  NE  on  the  012  degree  radial, 

maintain _ (altitude).  This  is  for  flow  control  into  CLT.  You  read  back  the  clearance  and  inform  your 

passengers  of  the  delay. 

What  are  your  actions  now?” 

(Expect  Further  Clearance  Time  1815,  Time  Now  1743) 

(When  finished,  give  heading  180,  vectors  for  Charotte) 

TASK  5  -  Windsheat  Recovery  on  Approach 

“You  are  now  30  miles  from  the  Charlotte  airport  at _ feet  and  just  received  the  ATIS  as  follows: 

“Charlotte-Douglas  International  Airport  information  Whiskey,  1950  special  weather,  sky  partially  obscured, 
visibility  one  and  one-half  miles,  thunderstorm,  rain-showers,  temperature  63,  dew-point  61,  wind  200  at  12, 
peak-gusts  25,  altimeter  29.83.  Expect  ILS  18L  or  ILS  18R.  Low-level  windshear  advisories  in  progress.  Notams, 
RWY  18L  glide-slope  out-of-service.  Advise  you  have  Whiskey”.  Indianapolis  center  tells  you  to  contact 
approach.  You  change  frequencies  and  are  given  a  vector  for  the  approach  to  18L  and  are  advised  that  a  USAir 
DC-9  reported  minus  15  knots  on  the  approach  to  18R. 

What  considerations  will  you  make  regarding  this  approach?” 

TASK  6  -  Abnormal  Gear  Extension 

“You  have  been  cleared  for  the  approach  to  RWY _ .  Following  your  normal  procedures,  you  put  the 

gear  handle  down,  and  discover  the  red  “gear  unsafe”  light(s)  are  still  illuminated. 

What  will  you  do  next?” 

(When  do  you  normally  put  the  gear  down?) 

(Gear  Lights:  Cessna  -  1  Red,  Piper,  2  Green  for  Main  Gear  No  Nosegear  indication,  Beechcraft  -  3  green  down- 
lock  lights,  1  red  “unsafe”) 


SUBTECT  DEBRIEFING 

Prompting  questions  for  each  segment: 

1 .  What  other  information  would  you  request? 

2.  Tell  me  how  you  came  to  that  conclusion. 

3.  What  factors  are  most  important  in  your  decision  to 
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4.  Could  you  tell  me  in  more  detail  why  you _ ? 

5.  Please  elaborate. 

POST  SESSION  DISCUSSION 

1.  Where  did  you  get  your  information/ training  in  these  areas? 

2.  Have  you  ever  been  in  similar  situations? 

a.  Which  ones? 

b.  Tell  me  about  your  experience. 

3.  Do  you  have  any  questions? 
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